Data processing method, end-side device, storage medium, chip system and computer program product

Abstract

The embodiment of the invention provides a data processing method, end side equipment, a storage medium, a chip system and a computer program product, and relates to the technical field of terminals. Due to the fact that the second expert modules in the first storage are formed in the mode that the second thread executes the preloading process based on the expert preference strategy of the first request under the condition that the first thread executes the prefilling stage process through the first model, parallel execution of the preloading process and the prefilling stage process is achieved, I / O delay is reduced, and the processing efficiency is improved. And the reasoning efficiency of the first model is improved. And because the first request has expert preference, and the plurality of second expert modules stored in the first memory are determined according to the first request, the first thread performs the decoding stage flow by using the first model, and the second thread performs the decoding stage flow by using the second model. The first thread can directly obtain the second expert module corresponding to the first expert module from the first storage, so that the cache hit rate is improved, and the reasoning efficiency of the first model is improved.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a data processing method, a terminal device, a storage medium, a chip system, and a computer program product. Background Art

[0002] With the development of terminal technology and artificial intelligence (AI) technology, large language models (LLMs) can be deployed on end-side devices to provide users with a better AI experience by taking advantage of the local computing, low latency, low power consumption, security, reliability, availability or privacy protection features of end-side devices. The large model deployed on the end-side device can be called an end-side model. The end-side model can be applied to at least one of natural language processing (NLP) tasks or computer vision (CV) tasks.

[0003] The inference efficiency of the client-side model is low, which affects the user experience. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, a terminal device, a storage medium, a chip system, and a computer program product, which are applied to the field of terminal technology and can improve the reasoning efficiency of a first model (i.e., a terminal model).

[0005] In a first aspect, an embodiment of the present application provides a data processing method. The method may include: when a first thread uses a first model to execute a second stage process, when the first thread has a second expert module corresponding to the first expert module in the first storage, the second expert module corresponding to the first expert module is obtained from the first storage. The first thread may use the second expert module corresponding to the first expert module to process the first word.

[0006] The first thread may execute the first stage process using the first model in parallel with the second thread executing the preloading process.

[0007] The first set stored in the first storage unit may be determined by the second thread according to the first request when the first thread executes the first stage process using the first model. The first set may include multiple second expert modules. The first stage process may be used to indicate that the first request is processed using the first model to obtain the second word element. The second word element may be used to indicate the first output word element corresponding to the first request. The second stage process may be used to indicate that the inference result is obtained according to the first request and the second word element using the first model. The first stage may also be called the pre-filling stage. The second stage may also be called the decoding stage. The pre-filling stage may refer to the stage of executing the first round of processing. The decoding stage may refer to the stage of executing the second to Pth rounds of processing. P may be an integer greater than 1. The first storage unit storing the second expert module may be understood as the first storage unit storing the model parameters of the second expert module.

[0008] The first word-element may be used to indicate an output word-element. For example, the first word-element may be the pth output word-element. The second word-element may be the 1st output word-element. The second word-element may also be referred to as the first word-element. p may be an integer greater than or equal to 2 and less than or equal to P.

[0009] The first expert module may be determined by the first thread from a plurality of expert modules included in the first model according to the first word corresponding to the first request. The first expert module may have a first identifier. The expert module may have a second identifier. The first identifier may be used to indicate the first expert. The second identifier may be used to indicate the expert module.

[0010] In an implementable manner, the first expert module may be determined by the first thread from the multiple expert modules included in the first model according to the third weights corresponding to the multiple expert modules included in the first model. The third weight may be obtained by the first thread calling the gating module included in the first model and using the gating module according to the first word corresponding to the first request.

[0011] According to the embodiment of the present application, since the multiple second expert modules of the first storage unit are determined by the second thread according to the first request when the first thread executes the first stage process using the first model, the parallel execution of the preloading process using the second thread and the first stage process using the first thread is realized, thereby hiding the I / O delay as much as possible, thereby reducing the I / O delay and improving the reasoning efficiency of the first model. And since the first request has an expert preference, and the multiple second expert modules stored in the first storage unit can be determined according to the first request, therefore, when the first thread executes the second stage process using the first model, the first thread can directly obtain the second expert module corresponding to the first expert module from the first storage unit, thereby improving the cache hit rate of the first thread obtaining the first expert module from the first storage unit, thereby improving the reasoning efficiency of the first model.

[0012] In an implementable manner, when the first thread uses the first model to execute the first stage process, the second thread can determine multiple third expert modules from the multiple expert modules included in the first model according to the first request. The second thread can determine the second expert modules corresponding to each of the multiple third expert modules from the second set. The second thread can retain the second expert modules corresponding to each of the multiple third expert modules in the second set to obtain the first set. That is, the first set stored in the first storage unit can be obtained by the second thread retaining the second expert modules corresponding to each of the multiple third expert modules in the second set when the first thread uses the first model to execute the first stage process. The multiple third expert modules can be determined by the second thread from the multiple expert modules included in the first model according to the first request. The second set can be stored in the first storage unit. The second set can include multiple second expert modules. The second set can be loaded from the second storage unit to the first storage unit by the first thread.

[0013] In the first stage (i.e., the pre-filling stage), the expert module in the activated state may be referred to as the second expert module. In the second stage (i.e., the decoding stage), the expert module in the activated state may be referred to as the first expert module. In the pre-loading stage, the expert module predicted to be in the activated state may be referred to as the third expert module. The second expert module may have a third identifier. The third expert module may have a fourth identifier. The third identifier may be used to indicate the second expert module. The fourth identifier may be used to indicate the third expert module. The second expert module corresponding to the third expert module may be understood as the second expert module being the third expert module.

[0014] The second storage unit can be used to store the expert module included in the first model. The second storage unit storing the expert module can be understood as the second storage unit storing the model parameters of the expert module.

[0015] The first speed may be greater than the second speed. The first speed may be used to indicate the speed at which the first thread accesses the first storage unit. The second speed may be used to indicate the speed at which the first thread accesses the second storage unit.

[0016] For any third expert module among the multiple third expert modules corresponding to the first request, in the process of the first thread loading at least one second expert module corresponding to at least one third word from the second storage unit to the first storage unit, if the second thread determines that there is a second expert module corresponding to the third expert module in the second expert modules stored in the first storage unit, that is, if it is determined that the third expert module exists in the second expert modules stored in the first storage unit, then the second expert module corresponding to the third expert module in the first storage unit can be retained.

[0017] In an implementable manner, the second thread may store the second expert module corresponding to the third expert module in the first storage unit into a queue.

[0018] If the second thread determines that there is no second expert module corresponding to the third expert module in the second set stored in the first storage unit, the expert module corresponding to the third expert module may be loaded from the second storage unit to the first storage unit. For example, the second thread may load the expert module corresponding to the third expert module from the second storage unit to the queue of the first storage unit.

[0019] Therefore, when the first thread executes the second stage process, the second expert module stored in the first storage unit corresponds to the third expert module, that is, the first storage unit stores the third expert module.

[0020] Since the first set stored in the first storage unit can be obtained by retaining the second expert modules corresponding to the third expert modules in the second set when the first thread executes the first stage process using the first model, the parallel execution of the preloading process using the second thread and the first stage process using the first thread is realized, thereby hiding the I / O delay as much as possible, thereby reducing the I / O delay and improving the reasoning efficiency of the first model. And since the first request has an expert preference, and the third expert module can be determined by the second thread according to the first request when the first thread executes the first stage process using the first model, therefore, when the first thread executes the second stage process using the first model, the first thread can directly obtain the second expert module corresponding to the first expert module from the first storage unit, thereby improving the cache hit rate of the first thread obtaining the first expert module from the first storage unit, thereby improving the reasoning efficiency of the first model.

[0021] In an implementable manner, the second thread determines a plurality of third expert modules from a plurality of expert modules according to the first request, which may include: the second thread may determine a plurality of third expert modules from a plurality of expert modules included in the first model using the second model according to the first request. The second model may be trained using the second request and a plurality of first scores corresponding to the second request. The first score may be used to indicate an activation score of the second expert module.

[0022] In one achievable manner, the first set stored in the first storage unit may be determined by the second thread according to the first request using the second model when the first thread executes the first stage process using the first model. For example, the first set stored in the first storage unit may be obtained by the second thread inputting the first request into the second model when the first thread executes the first stage process using the first model. The second model may be trained using the second request and a plurality of first scores corresponding to the second request. The first score may be used to indicate the activation score of the expert module.

[0023] In an implementable manner, the second thread may input the first request into the second model to obtain second scores corresponding to each of the plurality of expert modules. According to the second scores corresponding to each of the plurality of expert modules, a plurality of third expert modules corresponding to the first request are obtained. The second scores may be used to indicate the predicted activation scores of the expert modules. The predicted activation scores may be used to indicate the probability of being activated.

[0024] In one implementable manner, obtaining multiple third expert modules corresponding to the first request according to the second scores corresponding to each of the multiple expert modules may include: the second thread may determine U second scores from the second scores corresponding to each of the multiple expert modules. The expert modules corresponding to each of the U second scores are determined as the third expert modules. In the case where the probability of the expert module being accessed is agreed upon as the larger the second score is, the U second scores may be the maximum U second scores. In the case where the probability of the expert module being accessed is agreed as the smaller the second score is, the U second scores may be the minimum U second scores. U may be an integer greater than 1 and less than R*S. U may be determined according to the storage capacity of the first storage unit, which is not limited here.

[0025] In one implementable manner, the second model may be obtained by adjusting the model parameters of the second model according to the loss function value. The loss function value may be obtained based on the loss function according to the first scores and the second scores corresponding to the plurality of expert modules respectively. The second scores corresponding to the plurality of expert modules respectively may be obtained by inputting the second request into the second model. The second score may be used to indicate the predicted activation score of the expert module.

[0026] In one implementable manner, the loss function value may be obtained according to the loss function values ​​corresponding to each of the plurality of expert modules. The loss function value corresponding to the expert module may be obtained by inputting the first score and the second score corresponding to the expert module into the loss function.

[0027] Since the second model can be trained based on the expert preference of the request, that is, the second model can be trained using the second request and multiple first scores corresponding to the second request, the first score can be used to indicate the activation score of the expert module, thereby improving the expert prediction accuracy of the second model. On this basis, the first set stored in the first storage unit can be obtained by the second thread inputting the first request into the second model when the first thread executes the first stage process using the first model. Therefore, the first thread can directly obtain the second expert module corresponding to the first expert module from the first storage unit, thereby improving the cache hit rate of the first thread obtaining the first expert module from the first storage unit, thereby improving the reasoning efficiency of the first model.

[0028] In one achievable manner, the first thread may determine at least one third word-element according to the first request. The first thread may determine at least one second expert module corresponding to each of the at least one third word-element from the multiple expert modules included in the first model. The first thread may load at least one second expert module corresponding to each of the at least one third word-element from the second storage unit to the first storage unit to obtain a second set. That is, the second set may be the result of the first thread loading at least one second expert module corresponding to each of the at least one third word-element from the second storage unit to the first storage unit. The at least one second expert module corresponding to each of the at least one third word-element may be determined by the first thread according to the at least one third word-element from the multiple expert modules included in the first model. The at least one third word-element may be determined by the first thread according to the first request.

[0029] The first model may include at least one first layer. The first layer may include multiple expert modules. The at least one expert module corresponding to the third word may include at least one expert module corresponding to each of the at least one first layer. For any first layer in the at least one first layer, the first layer may be used as the current layer. For any third word in the at least one third word, the at least one expert module corresponding to the current layer and the third word may be determined by the first thread according to the third word from multiple expert modules corresponding to each of the at least one first layer included in the first model.

[0030] In an implementable manner, the first thread may obtain the second expert module corresponding to the fourth expert module from the first storage unit when the second expert module corresponding to the first expert module does not exist in the first storage unit and the second expert module corresponding to the fourth expert module exists in the first storage unit. The first thread may process the first word unit using the second expert module corresponding to the fourth expert module.

[0031] The first expert module may have a corresponding first weight. The first weight may be used to indicate the activation probability of the first expert module. The fourth expert module may have a corresponding second weight. The second weight may be used to indicate the activation probability of the fourth expert module. The absolute value of the difference between the first weight and the second weight may be less than or equal to the first threshold. Thus, the first weight may be greater than the second weight, or less than the second weight, as long as the absolute value of the difference between the first weight and the second weight is less than or equal to the first threshold. The first threshold may be configured according to actual business needs and is not limited here. For example, the first threshold may be less than or equal to 0.3.

[0032] The first thread may, if there is no second expert module corresponding to the first expert module in the first storage unit, obtain the second expert module corresponding to the fourth expert module from the first storage unit if there is a fourth expert module corresponding to the first expert module among the multiple expert modules included in the first model, if there is a second expert module corresponding to the fourth expert module in the first storage unit.

[0033] Since the absolute value of the difference between the second weight corresponding to the fourth expert module and the first weight corresponding to the first expert module is less than or equal to the first threshold, the second weight can be used to indicate the activation probability of the fourth expert module, and the first weight can be used to indicate the activation probability of the first expert module. Therefore, it can be shown that the prediction accuracy of the fourth expert module is close to the prediction accuracy of the first expert module, and thus, the fourth expert module can replace the first expert module. Based on this, when the second expert module corresponding to the first expert module does not exist in the first storage unit and the second expert module corresponding to the fourth expert module exists in the first storage unit, the first thread can directly obtain the second expert module corresponding to the fourth expert module from the first storage unit to use the second expert module corresponding to the fourth expert module to process the first word, thereby further improving the cache hit rate. In addition, since there is no need to first load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, and then obtain the expert module corresponding to the first expert module, and use the expert module corresponding to the first expert module to process the first word, the I / O delay between the second storage unit and the first storage unit is reduced, thereby improving the reasoning efficiency of the first model.

[0034] In an implementable manner, the first thread may call the gating module included in the first model, and use the gating module to obtain the third weight corresponding to each of the multiple expert modules according to the first word corresponding to the first request. The first thread may determine the first weight according to the third weight corresponding to each of the multiple expert modules. The first thread may determine the second weight from the third weights corresponding to each of the multiple expert modules according to the first weight and the first threshold, that is, the first weight may be determined by the first thread according to the third weight corresponding to each of the multiple expert modules included in the first model. The third weight corresponding to each of the multiple expert modules that may be included in the first model may be obtained by the first thread calling the gating module included in the first model and using the gating module according to the first word corresponding to the first request. The second weight may be determined by the first thread according to the first weight and the first threshold from the third weights corresponding to each of the multiple expert modules included in the first model.

[0035] Since the second weight corresponding to the fourth expert module can be determined from the third weights corresponding to the plurality of expert modules included in the first model according to the first threshold and the first weight corresponding to the first expert module, the second weight can be used to indicate the activation probability of the fourth expert module, and the first weight can be used to indicate the activation probability of the first expert module, thus achieving that the determined activation probability of the fourth expert module is closer to the activation probability of the first expert module. Thus, the fourth expert module can replace the first expert module to process the first word.

[0036] In an implementable manner, the fourth expert module may be the second expert module in the first set corresponding to the first request.

[0037] In another achievable manner, the first set may further include a plurality of second expert modules corresponding to the third request. The fourth expert module may be a second expert module in the first set corresponding to the third request. The third request may be different from the first request.

[0038] Since the fourth expert module can be the second expert module corresponding to the first request or the second expert module corresponding to the third request, the diversity of sources of the fourth expert module is improved, which helps to further reduce I / O delay.

[0039] In one implementable manner, the first thread may load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtain the expert module corresponding to the first expert module from the first storage unit, and process the first word using the expert module corresponding to the first expert module when the first storage unit does not contain the second expert module corresponding to the first expert module and the first storage unit does not contain the second expert module corresponding to the fourth expert module.

[0040] In one achievable manner, when the first storage unit does not contain the second expert module corresponding to the first expert module and the first storage unit does not contain the second expert module corresponding to the fourth expert module, the third thread obtains the expert module corresponding to the first expert module from the second storage unit and uses the expert module corresponding to the first expert module to process the first word. The first thread can be used to indicate an accelerator thread. The third thread can be used to indicate a processor thread.

[0041] Since the first thread can be used to indicate the accelerator thread, the third thread can be used to indicate the processor thread, the first storage unit can be used to indicate the accelerator memory, the second storage unit can be used to indicate the processor memory, the third speed can be used to indicate the speed at which the third thread accesses the second storage unit, and the fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit, the third access speed is greater than the fourth access speed. When the first thread determines that the first storage unit does not contain the second expert module corresponding to the first expert module and does not contain the second expert module corresponding to the fourth expert module, the third thread is used to obtain the expert module corresponding to the first expert module from the second storage unit, and the expert module corresponding to the first expert module is used to process the first word, without the first thread first loading the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtaining the expert module corresponding to the first expert module from the first storage unit, and then using the expert module corresponding to the first expert module to process the first word, thereby reducing the I / O delay between the second storage unit and the first storage unit and improving the reasoning efficiency of the first model.

[0042] In one implementable manner, the first thread may load an expert module corresponding to the first expert module from the second storage unit to the first storage unit if the second expert module corresponding to the first expert module does not exist in the first storage unit, obtain the expert module corresponding to the first expert module from the first storage unit, and use the expert module corresponding to the first expert module to process the first word.

[0043] In another achievable manner, when the first storage unit does not contain the second expert module corresponding to the first expert module, the third thread may obtain the expert module corresponding to the first expert module from the second storage unit, and use the expert module corresponding to the first expert module to process the first word. The third thread may be used to indicate the processor thread. The first thread may be used to indicate the accelerator thread.

[0044] Since the first thread can be used to indicate the accelerator thread, the third thread can be used to indicate the processor thread, the first storage unit can be used to indicate the accelerator memory, the second storage unit can be used to indicate the processor memory, the third speed can be used to indicate the speed at which the third thread accesses the second storage unit, and the fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit, the third access speed is greater than the fourth access speed. When the first thread determines that the first storage unit does not contain the second expert module corresponding to the first expert module, the third thread is used to obtain the expert module corresponding to the first expert module from the second storage unit, and the expert module corresponding to the first expert module is used to process the first word, without the first thread first loading the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtaining the expert module corresponding to the first expert module from the first storage unit, and then using the expert module corresponding to the first expert module to process the first word, thereby reducing the I / O delay between the second storage unit and the first storage unit and improving the reasoning efficiency of the first model.

[0045] The fourth thread may eliminate the second expert module from the first set based on the cache elimination strategy when the first storage unit satisfies the first condition. The first condition may be used to trigger cache elimination. The fourth thread may be executed in parallel with the first thread and the second thread.

[0046] Since the fourth thread can eliminate the second expert module from the first set based on the cache elimination strategy when the first storage unit meets the first condition, the fourth thread can be executed in parallel with the first thread and the second thread, thereby improving the cache hit rate and resource utilization when the storage capacity of the first storage unit is limited.

[0047] In one implementable manner, the first condition may include at least one of the following: the first capacity reaches the second threshold, the first number reaches the third threshold, the first usage rate reaches the fourth threshold, or the first hit rate is less than or equal to the fifth threshold. The first capacity can be used to indicate the total storage size of the first set. The second threshold can be determined based on the storage capacity of the first storage unit. The first number can be used to indicate the total number of second expert modules in the first set. The third threshold can be determined based on the storage capacity of the first storage unit and the storage size of the second expert module. The first usage rate can be used to indicate the ratio between the first capacity and the storage capacity of the first storage unit. The first hit rate can be used to indicate the ratio between the second number and the third number. The second number can be used to indicate the number of cache hits corresponding to the first storage unit. The third number can be used to indicate the total number of accesses corresponding to the first storage unit. And / or

[0048] The cache elimination policy may include at least one of the following: a first-in-first-out policy, a least recently used policy, a least frequently used policy, a most recently used policy, a random elimination policy, or a low interactive reference interval policy.

[0049] In an implementable manner, the first storage unit may be used to indicate an accelerator memory, and the second storage unit may be used to indicate a processor memory or a storage device.

[0050] In another possible implementation, the first storage unit may be used to indicate a processor memory, and the second storage unit may be used to indicate a storage device.

[0051] Since the second storage unit can be used to store multiple expert modules included in the first model, and the first storage unit can be used to store the expert modules of the activated experts, the cooperation between the second storage unit and the first storage unit alleviates the limited memory capacity of the end-side device, making it possible for the end-side device to deploy the first model.

[0052] In one implementation, the accelerator memory may be used to indicate a graphics processor memory, a neural network processor memory, or a tensor processor memory.

[0053] In one implementable manner, the third thread may be a processor thread. In the case where the first storage unit may be used to indicate the accelerator memory, the first thread and the second thread may be accelerator threads. The fourth thread may be an accelerator thread. In the case where the first storage unit may be used to indicate the processor memory, the first thread, the second thread, and the fourth thread may be processor threads.

[0054] In a second aspect, an embodiment of the present application provides a data processing device, which may include: an acquisition module, which is used for the first thread to acquire a second expert module corresponding to the first expert module from the first storage unit when the first thread executes the second stage process using the first model and the first thread has a second expert module corresponding to the first expert module in the first storage unit. The first expert module is determined by the first thread from multiple expert modules included in the first model according to the first word corresponding to the first request. A processing module is used for the first thread to process the first word using the second expert module corresponding to the first expert module.

[0055] According to an embodiment of the present application, the first set stored in the first storage unit may be determined by the second thread according to the first request when the first thread executes the first stage process using the first model. The first set may include multiple second expert modules. The first stage process may be used to indicate that the first request is processed using the first model to obtain the second word element. The second word element may be used to indicate the first output word element corresponding to the first request. The second stage process may be used to indicate that the inference result is obtained according to the first request and the second word element using the first model.

[0056] In a third aspect, an embodiment of the present application provides an end-side device, including a processor and a memory, the memory being used to store computer-executable instructions, and the processor being used to run the computer-executable instructions stored in the memory to execute the method described in the first aspect or any one of the implementable embodiments of the first aspect.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program or instruction is stored. When the computer program or instruction is executed on a computer, the terminal device executes the method described in the first aspect or any one of the implementable embodiments of the first aspect.

[0058] In a fifth aspect, an embodiment of the present application provides a computer program product including a computer program. When the computer program is executed, the terminal-side device executes the method described in the first aspect or any one implementable manner of the first aspect.

[0059] In a sixth aspect, the present application provides a chip or a chip system, the chip or chip system comprising at least one processor and a communication interface, the communication interface and the at least one processor are interconnected by a line, and the at least one processor is used to run a computer program or instruction to execute the method described in the first aspect or any one of the implementable embodiments of the first aspect. The communication interface in the chip can be an input / output interface, a pin or a circuit, etc.

[0060] In a possible implementation, the chip or chip system described above in the present application further includes at least one memory, in which instructions are stored. The memory may be a storage unit inside the chip, such as a register, a cache, etc., or a storage unit of the chip (such as a read-only memory, a random access memory, etc.).

[0061] It should be understood that the second to sixth aspects of the present application correspond to the technical solutions of the first aspect of the present application, and the beneficial effects achieved by each aspect and the corresponding feasible implementation methods are similar and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1A A schematic diagram of a converter provided for related art;

[0063] Figure 1B A schematic diagram of the coding layer provided for the related art;

[0064] Figure 1C A schematic diagram of a decoding layer provided for related art;

[0065] Figure 2A A schematic diagram of a large encoder-based model provided in an embodiment of the present application;

[0066] Figure 2B A schematic diagram of a large decoder-based model provided in an embodiment of the present application;

[0067] Figure 2C A schematic diagram of a large encoder-decoder model provided in an embodiment of the present application;

[0068] Figure 3 A schematic diagram of the application of the reasoning process of the large model provided in the embodiment of the present application;

[0069] Figure 4A A schematic diagram of a hybrid expert provided for an embodiment of the present application;

[0070] Figure 4B A schematic diagram of another hybrid expert provided for an embodiment of the present application;

[0071] Figure 4C A schematic diagram of an expert module provided in an embodiment of the present application;

[0072] Figure 5 A schematic diagram of the MoE model provided in the embodiments of the present application;

[0073] Figure 6 A schematic diagram of a first model provided in an embodiment of the present application;

[0074] Fig. 7A A schematic diagram of a storage structure of a terminal device provided in an embodiment of the present application;

[0075] Figure 7B A schematic diagram of a storage method of a first model provided in an embodiment of the present application;

[0076] Figure 7C A schematic diagram of another storage method of the first model provided in an embodiment of the present application;

[0077] Fig.7D A schematic diagram of another storage method of the first model provided in an embodiment of the present application;

[0078] Fig. 8A A schematic diagram of the principle of a data processing method provided in an embodiment of the present application;

[0079] Figure 8B A schematic diagram of another data processing method provided in an embodiment of the present application;

[0080] Figure 8C A schematic diagram of the principle of a request-based expert preference prediction method provided in an embodiment of the present application;

[0081] Fig.8D A schematic diagram of determining first scores corresponding to a plurality of second expert modules corresponding to a third word-gram provided in an embodiment of the present application;

[0082] Fig.9A A schematic diagram of replacing a first expert module with a fourth expert module provided in an embodiment of the present application;

[0083] Fig. 9B A schematic diagram of another embodiment of the present application in which a first expert module is replaced by a fourth expert module;

[0084] Fig.10 A schematic diagram of using a third thread to process a first word element provided in an embodiment of the present application;

[0085] Fig.11 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0086] Fig.12 A software structure diagram of a terminal device provided in an embodiment of the present application;

[0087] Fig.13 A flowchart of a data processing method provided in an embodiment of the present application;

[0088] Fig.14A A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application;

[0089] Fig. 14B A schematic diagram of storing a second expert module of a first model in a second storage provided in an embodiment of the present application;

[0090] Fig. 14C A schematic diagram of another data processing method provided in an embodiment of the present application;

[0091] Fig.14D A schematic diagram of a first thread using a first model to execute a first phase process and a second thread using a second model to execute a preloading process provided in an embodiment of the present application;

[0092] Fig.14E A schematic diagram of a first thread using a first model to execute a second phase process provided by an embodiment of the present application;

[0093] Fig.15A flowchart of another data processing method provided in an embodiment of the present application;

[0094] Fig.16 A flowchart of another data processing method provided in an embodiment of the present application;

[0095] Fig.17 A flowchart of another data processing method provided in an embodiment of the present application;

[0096] Fig.18 A flowchart of another data processing method provided in an embodiment of the present application;

[0097] Fig.19 This is a structural block diagram of a data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0098] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0099] To facilitate understanding of the embodiments of the present application, the following points are first explained.

[0100] 1. In the embodiments of the present application, "indication" may include direct indication, indirect indication, explicit indication or implicit indication. When describing that a certain indication information is used to indicate A, it can be understood that the indication information carries A, directly indicates A or indirectly indicates A.

[0101] The information indicated by the indication information may be referred to as information to be indicated. In the implementation process, there may be multiple ways to indicate the information to be indicated. For example, the information to be indicated itself may be directly indicated or the index of the information to be indicated may be indicated. Optionally, the information to be indicated may also be indirectly indicated by indicating other information. There may be an association between the other information and the information to be indicated. Optionally, a part of the information to be indicated may also be indicated, while the other part of the information to be indicated may be known or otherwise agreed upon. For example, the indication of information may be achieved by means of the arrangement order of multiple pieces of information agreed upon in advance (e.g., agreed upon by a protocol), thereby reducing the indication overhead to a certain extent. In addition, the common parts of multiple pieces of information may be identified and indicated uniformly to reduce the indication overhead caused by indicating the same information separately.

[0102] The indication method may also be various known at least methods. For example, it may be the above indication methods and various combinations thereof. The information to be indicated may exist in other equivalent forms, and the technical solutions provided in the embodiments of the present application should be understood to cover various forms.

[0103] The information to be indicated can be sent as a whole or divided into multiple sub-information and sent separately, and the sending period and / or sending time of the sub-information can be the same or different, which is not limited in the embodiment of the present application. The sending period and / or sending time of the sub-information can be predefined.

[0104] 2. In the embodiments of the present application, " / " can indicate that the objects associated with each other are in an "or" relationship. For example, "A / B" can indicate A or B. "And / or" can be used to describe that there are three kinds of relationships between the associated objects. For example, A and / or B can indicate that A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural.

[0105] 3. In the embodiments of the present application, "at least one" may refer to one or more. "Multiple" may refer to two or more, for example, two, three, four or more. Similar expressions (for example, at least one, at least one, etc.) apply in the same way. "At least one of the following", "one or more of the following" or similar expressions may refer to any combination of these items, and may include only a single item or a combination of plural items. For example, at least one of a, b or c may mean a, b or c; a and b; a and c; b and c; a, b and c. Among them, a, b, c may be singular or plural.

[0106] 4. In the embodiments of the present application, the various digital numbers involved are only used for the convenience of description and are not used to limit the scope of protection of the embodiments of the present application. The size of the serial number involved in the embodiments of the present application does not mean the order of execution, and the order of execution of each process should be determined by its function and internal logic. For example, the terms "first", "second", "third", "fourth" and other various terminology labels (if any) in the specification, claims and drawings of the embodiments of the present application can be used to distinguish similar objects without having to be used to describe a specific order or sequence. Among them, the terms used in this way can be interchangeable where appropriate, so that the embodiments described here can be implemented in an order other than what is illustrated or described here, and "first", "second", "third", "fourth", etc. are not necessarily different.

[0107] 5. In the embodiments of the present application, words such as "exemplary", "example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "example" or "for example" should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of "exemplary", "example" or "for example" is intended to present related concepts in a specific way for easy understanding.

[0108] 6. In the embodiments of the present application, the term "storage" or "saving" may refer to saving in one or more memories. The one or more memories may be separately provided or integrated in an encoder or decoder, a processor or a communication device. The one or more memories may also be partially separately provided and partially integrated in a decoder, a processor or a communication device. The type of memory may be any form of storage medium, which is not limited in the embodiments of the present application.

[0109] 7. In the embodiments of the present application, "including", "having" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to the steps or units that are clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, systems or devices.

[0110] 8. The arrows or boxes shown by dotted lines in the schematic diagrams of the drawings in the specification of the embodiments of the present application may represent optional steps or optional modules.

[0111] 9. Unless otherwise specified or there is no logical conflict, the terms and / or descriptions between different embodiments of the present application are consistent and can be referenced to each other. The technical features in different embodiments can be combined to form new embodiments according to their internal logical relationships.

[0112] In order to clearly describe the technical solutions of the embodiments of the present application, some terms involved in the embodiments of the present application are explained below.

[0113] 1. A request may be used to indicate a task to be processed by a large language model. The request may include task description data for describing the task. In addition, the request may also include prompting data. Prompting data may be used to assist the task description data so that the large language model can understand the request more accurately. Prompting data may include at least one of the following: context data, tasks, examples, format requirements, constraints, or prompting strategies.

[0114] The request may be an open request or a closed request, which is not limited in the present embodiment. For example, an open request may be "Please write a story". A closed request may be "In which city is FD University located?"

[0115] 2. Tokens can be the basic units for large language models (LLMs) to understand and process text. For example, a lemma can be a word, a subword, a stem, an affix, a single character, a phrase, a multi-word phrase, a punctuation mark, an identifier, or a number. A lemma can also be called a token or a tag.

[0116] The word-gram may include a word-gram having a predetermined function. For example, the predetermined function may include at least one of a start function or an end function. The start function may be used to indicate the start of a sequence. The end function may be used to indicate the end of a sequence. The word-gram used to indicate the start function may be referred to as a start word-gram. The word-gram used to indicate the end function may be referred to as an end word-gram. As an implementation, the start word-gram may be represented by [BOS]. The end word-gram may be represented by [EOS].

[0117] In the embodiment of the present application, the word element corresponding to the input data may be referred to as the input word element. The word element corresponding to the output result (or inference result) may be referred to as the output word element. The input data may be used to indicate the input of the large language model. The output result may be used to indicate the output of the large language model. The input data may be used to indicate the data included in the request.

[0118] 3. A word sequence may refer to at least one word arranged in order.

[0119] 4. Tokenization refers to the process of breaking down text into a sequence of tokens.

[0120] For example, "I like watching movies" can be tokenized to obtain a token sequence formed by the following multiple tokens ["I", "like", "watch", "movie"].

[0121] 5. Detokenization can refer to the process of recombining word sequences into a coherent text. Detokenization is the reverse process of lemmatization.

[0122] 6. A large language model can be a very large deep learning model that is pre-trained based on a large number of training samples. A large language model can have billions or even hundreds of billions of model parameters. A large language model can also be called a large-scale pre-trained language model, a large-scale language model, a large language model, or a large model. For the sake of brevity, the large language model will be referred to as a large model below.

[0123] The large model can be applied to at least one of the following: natural language processing tasks or computer vision tasks, etc. Natural language processing can be an interdisciplinary field covering multiple disciplines such as computer science, artificial intelligence and linguistics, aiming to study and develop algorithms and technologies that enable computers to understand, analyze, generate and process human language. Natural language processing may include natural language understanding (NLU) and natural language generation (NLG). Natural language understanding may refer to enabling computers to understand natural language. Natural language generation may refer to generating text in a natural language form that humans can understand based on structured data. The embodiments of this application do not limit the application of large language models. Computer vision may refer to enabling computers to understand and analyze visual information from images or videos.

[0124] As an implementation method, natural language processing tasks may include at least one of the following: intelligent question and answer, intelligent dialogue, machine translation, text generation, text summarization, speech synthesis, data analysis, programming assistance, document format conversion, information extraction, information retrieval, information recommendation, syntactic analysis, part-of-speech tagging, word segmentation, sentiment analysis, text classification or speech recognition, etc. For example, intelligent dialogue can be applied to at least one of the following: intelligent agent, intelligent assistant, chat robot, voice assistant, virtual assistant, smart home control, medical diagnosis, education assistance or social media interaction, etc.

[0125] As an implementation method, the computer vision task may include at least one of the following: visual classification, visual retrieval, object recognition, visual segmentation, visual question answering, image description, visual object detection, three-dimensional reconstruction, object tracking, cross-modal retrieval, multimodal matching, video summarization, emotion recognition or pedestrian re-identification, etc.

[0126] The large model architecture may include one of the following: a Transformer-based model architecture or a non-Transformer-based model architecture. The Transformer may include an encoder and a decoder. The encoder may be used to encode input data into an encoding vector. The encoding vector may be used to indicate semantic information of the input data. The decoder may be used to generate a complete output sequence based on the encoding vector and the generated output word sequence.

[0127] As an implementation, the Transformer-based model architecture may include at least one of the following: an encoder-only model architecture, a decoder-only model architecture, or an encoder-decoder-based model architecture. The decoder-based model architecture may include at least one of the following: a causal decoder (CD)-based model architecture or a prefix decoder (PD)-based model architecture. The large model based on the encoder-based model architecture may be referred to as an encoder-based large model. The large model based on the decoder-based model architecture may be referred to as a decoder-based large model. The large model based on the encoder-decoder model architecture may be referred to as an encoder-decoder-based large model. The decoder-based large model may use an autoregressive (AR) method to gradually generate an output sequence.

[0128] It should be noted that the model architecture of the large model described in the embodiment of the present application can be a model architecture based on Transformer. The model architecture based on Transformer can be a model architecture based on a standard Transformer, or a model architecture based on a non-standard Transformer, which is not limited in the embodiment of the present application. The model architecture of the non-standard Transformer can be obtained by improving the model architecture of the standard Transformer. For example, the encoder of the standard Transformer can be improved, and the decoder of the standard Transformer can also be improved.

[0129] For ease of understanding, the following Figure 1A-Figure 1C The Transformer is described below. Figure 1A-Figure 1C Based on Figure 2A-2C A large model based on the Transformer model architecture is explained.

[0130] Figure 1A A schematic diagram of a converter provided for related art.

[0131] like Figure 1A As shown, the converter 100 may include an encoder 110 and a decoder 120 .

[0132] The encoder 110 may include N cascaded encoding layers, namely, encoding layers 110_1, ..., encoding layers 110_n, ..., encoding layers 110_N. The decoder 120 may include N cascaded decoding layers, namely, decoding layers 120_1, ..., decoding layers 120_n, ..., decoding layers 120_N. The decoder 120 may generate a complete output sequence based on the generated output word sequence and the encoding vector output by the encoder 120_N. N may be an integer greater than or equal to 1. n may be an integer greater than or equal to 1 and less than or equal to N.

[0133] exist Figure 1A Based on the above, the following takes the encoding layer 110_n and the decoding layer 120_n as an example, combined with Figure 1B For the coding layer and Figure 1C The decoding layer is described.

[0134] Figure 1B A schematic diagram of the coding layer provided for the related art.

[0135] like Figure 1B As shown, the encoding layer 110_n may include a cascaded attention layer 111_n, a layer normalization (LN) 112_n, a feed-forward neural network (FFN) layer 113_n, and a layer normalization 114_n. In addition, the encoding layer 110_n may also include a residual connection between the input and output of the attention layer 111_n and a residual connection between the input and output of the feed-forward neural network layer 113_n. The residual connection may refer to connecting the input of the corresponding layer to the output of the corresponding layer using a direct connection channel.

[0136] The attention layer 111_n can be used to extract global features of the input. As an implementation, the attention layer 111_n can include a multi-head self-attention (MHA) layer. The feedforward neural network layer 113_n can be used to perform nonlinear transformation on the input and / or extract higher-level feature representations. As an implementation, the feedforward neural network layer 113_n can include a cascaded linear layer 1131_n, a first nonlinear activation and a linear layer 1132_n. The linear layer 1131_n can be used to increase the dimension of the input. The thread layer 1132_n can be used to reduce the dimension of the features after the dimension increase. The first nonlinear activation function can be used to perform nonlinear transformation on the input. The first nonlinear activation function can include a rectified linear unit (ReLU) function, a Swish function, a Sigmoid function, a Gaussian error linear unit (GELU) function, a gated linear unit (GLU) function, a Swiglu function or a Geglu function, etc.

[0137] Figure 1C A schematic diagram of a decoding layer provided for related art.

[0138] like Figure 1C As shown, the decoding layer 120_n may include an attention layer 121_n, a layer normalization 122_n, an attention layer 123_n, a layer normalization 124_n, a feedforward neural network layer 125_n, and a layer normalization 126_n. In addition, the decoding layer 120_n may also include a residual connection between the input and output of the attention layer 121_n, a residual connection between the input and output of the attention layer 123_n, and a residual connection between the input and output of the feedforward neural network layer 125_n.

[0139] The attention layer 121_n may include a masked multi-head attention (MMHA) layer. The attention layer 123_n may include a cross-attention (CA) layer. The input of the attention layer 123_n may include Figure 1A The output of the coding layer 110_N. As an implementation method, the model structure of the feedforward neural network layer 125_n can be the same as Figure 1B The feedforward neural network layer 113_n is similar and will not be described again here.

[0140] Figure 2A A schematic diagram of a large encoder-based model provided in an embodiment of the present application.

[0141] like Figure 2A As shown, the encoder-based large model may include cascaded coding layers 210_1, ..., coding layers 210_m, ..., coding layers 210_M. As an implementation, the model structure of the coding layer 210_m may be the same as Figure 1A The middle coding layer 110_n is similar and will not be described again. M may be an integer greater than or equal to 1. m may be an integer greater than or equal to 1 and less than or equal to M.

[0142] Figure 2B A schematic diagram of a large decoder-based model provided for an embodiment of the present application.

[0143] like Figure 2B As shown, the decoder-based large model may include cascaded decoding layers 220_1, ..., decoding layers 220_m, ..., decoding layers 220_M. The decoding layers included in the decoder-based large model are described below using the decoding layer 220_m as an example.

[0144] According to the configuration position of the normalization layer (NL), the model structure of the decoding layer 220_m can be implemented as follows.

[0145] The first method is based on post-layer normalization (Post-Norm), that is, the normalization layer can be configured after the residual connection. The decoding layer 220_m may include a cascaded attention layer 2201_m, a normalization layer 2202_m, a feedforward neural network layer 2203_m and a normalization layer 2204_m. In addition, the decoding layer 220_m may also include a residual connection between the input and output of the attention layer 2201_m and a residual connection between the input and output of the feedforward neural network layer 2203_m.

[0146] As an implementation manner, the normalization layer may include at least one of the following: layer normalization, root mean square layer normalization (RMSNorm), or DeepNorm, etc.

[0147] The second method is based on pre-layer normalization (Pre-Norm), that is, the normalization layer can be configured before the attention layer or the feedforward neural network layer. The decoding layer 220_m may include a cascaded normalization layer 2205_m, an attention layer 2206_m, a normalization layer 2207_m, and a feedforward neural network layer 2208_m. In addition, the decoding layer 220_m may also include a residual connection between the input of the normalization layer 2205_m and the output of the attention layer 2206_m and a residual connection between the input of the normalization layer 2207_m and the output of the feedforward neural network layer 2208_m.

[0148] The third method is based on sandwich normalization (Sandwich-Norm), that is, based on the pre-layer normalization, a normalization layer is added after the residual connection. The decoding layer 220_m may include a cascaded normalization layer 2209_m, an attention layer 2210_m, a normalization layer 2211_m, a normalization layer 2212_m, a feedforward neural network layer 2213_m, and a normalization layer 2214_m. In addition, the decoding layer 220_m may also include a residual connection between the input of the normalization layer 2209_m and the output of the normalization layer 2211_m and a residual connection between the input of the normalization layer 2212_m and the output of the normalization layer 2214_m.

[0149] Figure 2C A schematic diagram of a large encoder-decoder model provided in an embodiment of the present application.

[0150] like Figure 2C As shown, the encoder-decoder based large model may include an encoder 210 and a decoder 220. The encoder 210 may include cascaded encoding layers 210_1, ..., encoding layers 210_m, ..., encoding layers 210_M. The decoder 220 may include cascaded decoding layers 220_1, ..., decoding layers 220_m, ..., decoding layers 220_M. As an implementation, the model structure of the encoding layer 210_m may be the same as Figure 1A Similar to the coding layer 110_n, the model structure of the decoding layer 220_m can be the same as Figure 2B Similar, no further elaboration is required here.

[0151] 7. The reasoning stage of the large model may refer to the stage of using the large model to process the input data for P rounds to obtain the output result. P may be an integer greater than or equal to 1. P may be determined according to the end condition. The end condition may include at least one of the following: reaching the maximum sequence length, outputting the end word, or determining that a logical output result has been obtained. Thus, P may be the round corresponding to obtaining the maximum sequence length, the round corresponding to obtaining the end word, or the round corresponding to obtaining a logical output result.

[0152] As an implementation method, the reasoning stage of the large model may include a prefill stage and a decoding stage. The prefill stage may refer to the stage of performing the first round of processing. In the prefill stage, the input data may be processed by the large model to obtain the first output word and the intermediate state corresponding to the input data. The first output word may refer to the first output word. The intermediate state may include a key (i.e., Key) and (i.e., Value). The decoding stage may refer to the stage of performing the second to Pth rounds of processing. In the decoding stage, the input data, the first output word, and the intermediate state corresponding to the input data may be processed by the large model based on an autoregressive method to obtain the output result.

[0153] The pre-filling stage and the decoding stage are further described below.

[0154] In the pre-filling stage, when p=1, the large model can be used to process at least one input word in parallel to obtain a first output word and an intermediate state corresponding to each of the at least one input word. The at least one first input word can be obtained by tokenizing the input data.

[0155] In the decoding stage, that is, when 1<p≤P, the large model is used to obtain the pth output word and the intermediate state corresponding to the pth word according to at least one input word, the first output word to the p-1th output word, the intermediate state corresponding to each of the at least one input word, and the intermediate state corresponding to each of the first output word to the p-1th output word. Thus, the second output word to the Pth output word are obtained.

[0156] Based on the above, the output result is obtained according to the 1st output word unit to the Pth output word unit.

[0157] It should be noted that the pre-filling stage may also be referred to as the pre-processing stage, the initialization stage, the prompting stage, the full decoding stage, the full reasoning stage, the full stage or the context stage, etc. The decoding stage may also be referred to as the generating stage, the incremental decoding stage or the incremental stage, etc. In the embodiment of the present application, the first stage may be used to indicate the pre-filling stage. The second stage may be used to indicate the decoding stage.

[0158] In order to facilitate the understanding of the reasoning stage of the large model, the following Figure 3 Provide explanation.

[0159] Figure 3 A schematic diagram of the application of the reasoning process of the large model provided in the embodiment of the present application.

[0160] like Figure 3 As shown, the input data is "in which city is FD University located". The input data is processed by the large model for P rounds to obtain the output result. P=5.

[0161] Regarding how to use the large model to obtain output results, it can be achieved in the following ways.

[0162] In the pre-filling stage, that is, when p=1, "FD University is located in which city" is tokenized to obtain the following multiple input tokens, namely ["FD"], ["university"], ["located in"], ["which"], and ["city"]. The large model processes the multiple input tokens in parallel to obtain the first output token ["FD"] and the intermediate state corresponding to the first output token ["FD"].

[0163] In the decoding stage, that is, when 1<p≤P, the large model is used to obtain the pth output word and the intermediate state corresponding to the pth word according to multiple input words, the 1st output word to the p-1th output word, the intermediate state corresponding to at least one input word, and the intermediate state corresponding to the 1st output word to the p-1th output word. Thus, the 2nd output word to the Pth output word are obtained. The 2nd output word is [“University”]. The 3rd output word is [“Located in”]. The 4th output word is [“SH”]. The 5th output word is [“EOS”].

[0164] Based on the above, according to the 1st output word unit to the Pth output word unit, the output result "FD University is located in SH" is obtained.

[0165] It should be noted that Figure 3 Intermediate states are not shown.

[0166] 8. Parameter sparsity can mean that only some model parameters are involved in task processing.

[0167] 9. Mixture of experts (MoE) is a model architecture that participates in task processing by dynamically selecting some model parameters, which can improve computational efficiency and model performance.

[0168] The hybrid expert may include at least one gating module and multiple expert modules. The model structures of the multiple expert modules may be the same, but the model parameters of the multiple expert modules are different. Optionally, the model structures of the multiple expert modules may also be different. Different expert modules may learn different feature representations. The hybrid expert may assign word units to the expert modules in the activated state for processing through the gating module. The expert module in the activated state may refer to the expert module selected by the gating module. The expert module in the inactivated state may refer to the expert module not selected by the gating module. As a result, the hybrid expert has parameter sparsity.

[0169] The gating module can be used to generate weights corresponding to each of the multiple expert modules according to the word element. The weights corresponding to the expert modules can be used to indicate the degree of contribution of the expert modules to the output results. The weights can be expressed by activation probabilities. Thus, the gating module can determine at least one target expert module from the multiple expert modules according to the weights corresponding to each of the multiple expert modules. The target expert module can refer to an expert module in an activated state. The target expert module can be used to process word elements. The model structure of the gating module can be configured according to actual business needs, and the embodiments of the present application are not limited to this. As an implementation method, the gating module may include a cascaded linear layer and a third nonlinear activation function. The gating module may also be referred to as a routing module or a router.

[0170] The expert module can be used to perform nonlinear transformation and / or feature extraction on the input. As an implementation method, multiple expert modules can all be non-shared expert modules. Optionally, multiple expert modules can include at least one shared expert module and multiple non-shared expert modules. The non-shared expert module can refer to an expert module for processing word units when activated. The shared expert module can be used to cooperate with the non-shared expert module to process the expert module of the word unit. In the case where multiple expert modules include at least one shared expert module and multiple non-shared expert modules, the gating module can be used to determine at least one target expert module from multiple non-shared expert modules. The model structure of the expert module can be configured according to actual business needs, and the embodiment of the present application does not limit this. As an implementation method, the expert module can be similar to a feedforward neural network layer. The number of target expert modules can be configured according to actual business needs, and the embodiment of the present application does not limit this. For example, the number of target expert modules can be 2.

[0171] Thus, when multiple expert modules are non-shared expert modules, the gating module can determine the weights corresponding to the multiple expert modules according to the word element, and determine at least one target expert module from the multiple expert modules according to the weights corresponding to the multiple experts. The target expert module can obtain the output result corresponding to the target expert module according to the word element. The hybrid expert can obtain the output result corresponding to the word element according to the weight corresponding to the at least one target expert module and the output result corresponding to the at least one target expert module.

[0172] In the case where the multiple expert modules include at least one shared expert module and multiple non-shared expert modules, the gating module can determine the weights corresponding to the multiple non-shared expert modules according to the word element, and determine at least one target expert module from the multiple non-shared expert modules according to the weights corresponding to the multiple non-shared expert modules. The module expert module obtains the output result corresponding to the target expert module according to the word element. The shared expert module can obtain the output result corresponding to the shared expert module according to the word element. The hybrid expert can obtain the output result corresponding to the word element according to the weight corresponding to each of the at least one non-shared expert modules, the output result corresponding to each of the at least one non-shared expert modules, and the output result corresponding to the shared expert module.

[0173] For ease of understanding, the following Figure 4A-4C Explain the hybrid expert. Figure 4A Multiple expert modules are non-shared expert modules. Figure 4B The multiple expert modules include a shared expert module and multiple non-shared expert modules. Figure 4C Shows Figure 4A and Figure 4B Model structure of the expert module in .

[0174] Figure 4A A schematic diagram of a hybrid expert provided in an embodiment of the present application.

[0175] like Figure 4A As shown, the hybrid expert 400 may include a gating module 410 and Q expert modules. The Q expert modules may include expert modules 420_1, ..., expert modules 420_q, ..., expert modules 420_Q. Q may be an integer greater than 1, for example, Q=8. q may be an integer greater than or equal to 1 and less than or equal to Q.

[0176] The gating module can determine the weights corresponding to the Q expert modules according to the word element, and determine K target expert modules from the Q expert modules according to the weights corresponding to the Q experts. K=2. For example, K weights can be determined from the Q weights. The expert modules corresponding to the K weights are determined as the target expert modules. Any weight among the K weights is greater than any weight among the Q weights except the K weights. Since the weight corresponding to the expert module 420_1 and the weight corresponding to the expert module 420_q are both greater than any weight among the Q weights except the two weights, the expert module 420_1 and the expert module 420_q are the target expert modules. q=4.

[0177] Based on this, the expert module 420_1 obtains the output result corresponding to the expert module 420_1 according to the word-gram. The expert module 420_q obtains the output result corresponding to the expert module 420_q according to the word-gram. The hybrid expert 400 obtains the weighted result corresponding to the expert module 420_1 according to the weight and output result corresponding to the expert module 420_1. The hybrid expert 400 obtains the weighted result corresponding to the expert module 420_q according to the weight and output result corresponding to the expert module 420_q. The hybrid expert 400 obtains the output result corresponding to the word-gram according to the weighted results corresponding to the expert module 420_1 and the expert module 420_q respectively.

[0178] Figure 4B A schematic diagram of another hybrid expert provided in an embodiment of the present application.

[0179] like Figure 4B As shown, the hybrid expert 400 may include a gating module 410 and Q+1 expert modules. The Q+1 expert modules may include a shared expert module 420_0 and Q non-shared expert modules. The Q non-shared expert modules may include expert modules 420_1, ..., expert modules 420_q, ..., expert modules 420_Q. Q may be an integer greater than 1, for example, Q=8. q may be an integer greater than or equal to 1 and less than or equal to Q.

[0180] The gate control module can be used with Figure 4A In a similar manner, expert module 420_1 and expert module 420_q are determined as target expert modules from among the Q expert modules, which will not be described in detail herein.

[0181] Based on this, the shared expert module 420_0 can obtain the output result corresponding to the shared expert module 420_0 according to the word unit. The hybrid expert 400 obtains the output result corresponding to the word unit according to the weighted results corresponding to the expert module 420_1 and the expert module 420_q, and the output result corresponding to the shared expert module. Figure 4AIn a similar manner, weighted results corresponding to the expert module 420_1 and the expert module 420_q are determined, which will not be described in detail here.

[0182] Figure 4C A schematic diagram of an expert module provided in an embodiment of the present application.

[0183] like Figure 4C As shown, the model structure of the expert module 420_q can be implemented as follows.

[0184] In a first manner, the expert module 420_q may include a cascaded linear layer 4201_q, a first nonlinear activation function, and a linear layer 4202_q. For the description of the linear layer 4201_q, the linear layer 4202_q, and the first nonlinear activation function, refer to the description of the feedforward neural network layer 113_n above, which will not be repeated here.

[0185] In a second manner, the expert module 420_q may include a linear layer 4203_q, a linear layer 4204_q and a second non-linear activation function. The second non-linear activation function may include SwiGLU.

[0186] 10. The MoE model may refer to a large model obtained by replacing at least one feedforward neural network layer in the large model with a mixed expert layer (i.e., mixed expert). For example, the MoE model may include at least one mixed expert layer but does not include a feedforward neural network layer. Optionally, the MoE model may include at least one mixed expert layer and at least one feedforward neural network layer, which is not limited in this embodiment of the present application. The large model may be the decoder-based large model or the encoder-decoder-based large model described above. The model structures of different mixed expert layers may be the same or different, which is not limited in this embodiment of the present application.

[0187] The MoE model can expand the model size by increasing the number of expert modules, thereby improving model performance. Since the mixed expert layer has parameter sparsity, the MoE model can reduce computational overhead and improve computational efficiency (or reasoning efficiency) compared to dense models of the same size. In addition, since multiple expert modules are independent of each other, multiple expert modules can be calculated in parallel if the hardware supports it, thereby further improving computational efficiency. Since different expert modules can focus on different features, the model performance of the MoE model can be improved.

[0188] For ease of understanding, the following Figure 5 Provide explanation.

[0189] Figure 5 A schematic diagram of the MoE model provided in an embodiment of the present application.

[0190] like Figure 5As shown, the feedforward neural network layer in the large model can be replaced by the hybrid expert layer to obtain the MoE model. The description of the hybrid expert layer can be found in the corresponding part above, which will not be repeated here.

[0191] 11. The end-side device may refer to a device deployed near the data source. The end-side device is able to process local data without relying on cloud devices. The end-side device has at least one of the characteristics of local computing, low latency, low power consumption, reliability, availability or privacy protection. In addition, the resources of the end-side device are limited. Resources may include computing resources and storage resources, etc. The end-side device may also be called terminal equipment (TE), user equipment (UE), edge device, mobile device, mobile terminal (MT), user terminal, terminal, access terminal, remote terminal, user agent, user device or mobile station (MS), etc.

[0192] The end-side device may include at least one of the following: a mobile phone, a tablet computer, a portable computer, a desktop computer, an end-side device in autonomous driving (AD), an end-side device in the Internet of Things (IoT), an end-side device in vehicle-to-everything (V2X), an in-vehicle terminal device, a mobile Internet device (MID), a virtual reality (VR) device, an augmented reality (AR) device, a mixed reality (MR) device, a personal digital assistant (PDA), a customer premises equipment (CPE), a drone, a helicopter, an airplane, a ship, a robot, a robotic arm, an end-side device in device-to-device communication (D2D), an end-side device in industrial control (IC), and a machine-type communication (machine-type The terminal side devices in the mobile communication (MTC), the terminal side devices in the remote medical (RM), the terminal side devices in the smart grid (SG), the terminal side devices in the transportation safety (TS), the terminal side devices in the intelligent transportation, the terminal side devices in the smart city (SC), the terminal side devices in the smart home (SH), the terminal side devices in the smart office or the wearable devices, etc., are not limited to the embodiments of the present application.

[0193] Wearable devices are a general term for wearable devices developed by applying wearable technology to intelligently design daily wearables. For example, wearable devices may include at least one of the following: head mounted display (HMD), glasses, gloves, watches, clothing or shoes. Wearable devices are portable devices that are worn directly on the body or integrated into the user's clothes or accessories. Wearable devices can achieve powerful functions through software support, data interaction and cloud interaction. Broadly speaking, wearable devices include devices with full functions, large sizes or that do not rely on smartphones to achieve full or partial functions, such as smart watches or smart glasses, as well as devices that only focus on a certain type of application function and need to be used in conjunction with other devices such as smartphones, such as various types of smart bracelets or smart jewelry for vital sign monitoring. For example, smart glasses may include at least one of the following: VR glasses or AR glasses. Transmittable devices can also be called wearable smart devices.

[0194] 12. The end-side model may refer to a large model deployed on the end-side device. The end-side model may also be called a terminal model, a mobile end model, a client model, a device end model, a local model, an edge model or an embedded model. The end-side model has at least one of the characteristics of low latency, bandwidth saving, privacy protection, security, reliability, availability or personalization. The end-side model can be applied to at least one of natural language processing tasks or computer vision tasks. For the description of natural language processing tasks and computer vision tasks, please refer to the corresponding parts above, which will not be repeated here. In the embodiment of the present application, the first model may refer to the end-side model.

[0195] 13. An accelerator may refer to a hardware device specially designed to accelerate a predetermined computing task. An accelerator may work in conjunction with a general-purpose processor. For example, a general-purpose processor may include a central processing unit (CPU).

[0196] The accelerator may include at least one of the following: a graphics processing unit (GPU), a neural processing unit (NPU), a tensor processing unit (TPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). The GPU has a large number of GPU cores and can be used for high-parallel computing. In the embodiment of the present application, the accelerator may also be referred to as a hardware accelerator.

[0197] 14. The storage structure of the end-side device may include memory and storage devices. The memory may include at least one of the following: processor memory or accelerator memory. The accelerator memory may include at least one of the following: graphics processor memory, neural network processor memory, or tensor processor memory, etc. The storage device may include at least one of the following: flash memory (FM), embedded multimedia card (eMMC), universal flash storage (UFS), solid state drive (SSD) or hard desk drive (HDD), etc.

[0198] 15. Cache may refer to an area in memory for storing expert parameters. Expert parameters may be used to indicate model parameters of an expert module. In the embodiment of the present application, the model parameters of an expert module may be represented by an expert module. Thus, cache may also refer to an area in memory for storing expert modules.

[0199] 16. A cache hit (CH) may refer to a situation where the target expert module exists in the memory. A cache miss (CM) may refer to a situation where the target expert module does not exist in the memory. A target expert module may refer to an expert module in an activated state.

[0200] 17. Memory access speed may refer to the time taken by a processor or accelerator to access memory. Memory access speed may be affected by at least one of memory latency, memory bandwidth, memory frequency, memory type, or memory channel. Memory frequency may refer to the clock frequency of the memory.

[0201] 18. I / O latency can refer to the time it takes from issuing an input / output (i.e., Input / Output) request to receiving a response to the request. For example, I / O latency can include the time it takes for the accelerator memory to be retrieved from the processor memory or storage device to the expert module.

[0202] 19. An I / O bottleneck can refer to a situation where input / output operations become a performance limiting factor. I / O delays can cause an I / O bottleneck.

[0203] 20. Parallelism can refer to the ability to execute multiple tasks at the same time.

[0204] 21. Thread is the smallest execution and scheduling unit that can be recognized by the operating system. Threads can execute tasks independently. For example, multiple threads can be executed in parallel using a multi-core processor. Alternatively, multiple threads can be executed in parallel using multiple processors.

[0205] 22. Pipeline parallelism is a method of parallel computing that improves the execution efficiency of large models. For example, I / O computational pipelining can refer to a method of improving system performance by overlapping input / output operations with computations.

[0206] 23. A linked list (LL) is a linear data structure that may include at least one node. A node may include data and at least one pointer. The pointer may be used to point to a target node. A linked list may include at least one item in a one-way linked list or a two-way linked list. In the case where the linked list is a one-way linked list, the node may include data and a pointer. The target node may refer to the next node. In the case where the linked list is a two-way linked list, the node may include data and two pointers. The target node may include a previous node or a next node.

[0207] 24. An array is a linear data structure that may include a set of continuous memory space for storing elements of the same type.

[0208] 25. A queue is a linear list that allows insertion at one end and removal at the other end, following the first in first out (FIFO) principle. The end that allows insertion can be called the tail of the queue. The end that allows removal can be called the head of the queue. The data in the queue can be called queue elements. Continuous storage units can be allocated to store queue elements, and pointers can be used to indicate the head and tail elements. Queues can be implemented using arrays or linked lists.

[0209] 26. A multi-core processor can refer to a processor that concentrates multiple cores on the same chip to provide external services. A multi-core processor can also be called an on-chip multiprocessor or a single-chip multiprocessor.

[0210] A multi-core processor may include multiple computing cores, which can integrate multiple single-threaded computing cores or multiple multi-threaded computing cores. Multiple cores are deployed in the same chip, which can shorten the connection between cores, reduce the communication delay between cores, and improve communication efficiency. At the same time, multiple cores share on-chip resources, thereby improving the utilization of on-chip resources and reducing the power consumption of computing cores.

[0211] 27. Cache eviction policy (CEP) can refer to a strategy for determining the data to be eliminated (or removed) from the cache in order to provide cache space for new data when the cache space is limited.

[0212] The cache elimination policy may include at least one of the following: first-in-first-out policy, least recently used (LRU) policy, LRU-K policy, least frequently used (LFU) policy, most recently used (MRU) policy, random replacement (RR) policy, time to live (TTL) policy, two-queue (2Q) policy, adaptive replacement cache (ARC) policy, or low inter-reference recency set (LIRS) policy.

[0213] The first-in-first-out strategy can be used to prioritize the elimination of data that entered the cache the earliest. The least recently used strategy can be used to prioritize the elimination of data that has not been accessed for the longest time. The LRU-K strategy can be used to eliminate the data that has not been accessed for the longest time from the most recent K accesses. The least frequently used strategy can be used to prioritize the elimination of data with the lowest access frequency. The most recently used strategy can be used to prioritize the elimination of recently accessed data. The random elimination strategy can be used to randomly select data to be eliminated. The survival time strategy can be used to eliminate data that exceeds the life cycle corresponding to the storage item. The secondary cache strategy can be used to determine the eliminated data based on the most recently used data and the frequently accessed data. The adaptive replacement cache strategy can be used to dynamically adjust the allocation of cache space. The low interactive reference interval strategy can be used to optimize cache elimination based on the low interactive reference interval.

[0214] The inventive concept of the embodiments of the present application will be described below with reference to the accompanying drawings.

[0215] With the development of terminal technology and artificial intelligence technology, large models can be deployed on end-side devices to provide users with a better AI experience by taking advantage of at least one of the characteristics of local computing, low latency, low power consumption, security, reliability, availability or privacy protection of end-side devices. The large model deployed on the end-side device can be called an end-side model. The end-side model can be applied to at least one of natural language processing tasks or computer vision tasks.

[0216] Since the model parameters of a large model are large, it has high storage and computing requirements. However, due to the limited storage and computing resources of the end-side device, it is difficult to deploy a large model on the end-side device. In order to obtain a large model that can be deployed on the end-side device (i.e., the end-side model), it is necessary to consider how to adapt the large model to the end-side device.

[0217] Since the MoE model can reduce computing overhead and improve model performance, the MoE model can be used as a terminal-side model. However, there is a large gap between the memory requirements of the MoE model and the memory capacity of the terminal-side device. Therefore, it is necessary to consider how to deploy the MoE model on the terminal-side device.

[0218] In order to better understand the embodiments of the present application, Figure 6 The end-side model described in the embodiment of the present application is described. In the embodiment of the present application, the end-side model may be referred to as a first model. The first model may be an MoE model. The first model may include multiple first layers in cascade. The first layer may include a hybrid expert layer. The hybrid expert layer may include a gating module and multiple expert modules. For example, the first model may include R first layers in cascade. The hybrid expert layer may include S expert modules. R may be an integer greater than or equal to 1. S may be an integer greater than 1. The first layer currently being used may be referred to as the current layer.

[0219] Figure 6 A schematic diagram of the first model provided in an embodiment of the present application.

[0220] like Figure 6 As shown, the first model 600 may include R cascaded first layers, namely, first layer 600_1, ..., first layer 600_r, ..., first layer 600_R. r may be an integer greater than or equal to 1 and less than or equal to R. The model structures of different first layers may be the same or different, and the embodiment of the present application is not limited to this. For example, the R first layers may all include a mixed expert layer. Optionally, some of the R first layers may include a mixed expert layer, and another part of the first layers may include a feedforward neural network layer. The model structures of different mixed expert layers may be the same or different, and the embodiment of the present application is not limited to this.

[0221] The first layer 600_r is taken as an example to describe the first layer below. The first layer 600_r can be understood as the current layer.

[0222] The first layer 600_r may include a hybrid expert layer 610_r. The hybrid expert layer 610_r may include a gating module 611_r and S expert modules. The S expert modules may include an expert module 612_r_1, ..., an expert module 612_r_s, ..., an expert module 612_r_S. s may be an integer greater than or equal to 1 and less than or equal to S.

[0223] The gating module 611_r can be used to determine at least one first expert module from the expert modules 612_r_1, ..., expert modules 612_r_s, ..., expert modules 612_r_S. The first expert module may refer to an expert module in an activated state. The first expert module may be used to process input (e.g., word units). The number of first expert modules may be configured according to actual business needs and is not limited here. For example, the number of first expert modules may be 2. For other descriptions of the hybrid expert layer 610_r, please refer to the above description of the hybrid expert layer, which will not be repeated here.

[0224] In addition, the first layer 600_r may also include an attention layer and a normalization layer. For the description of the attention layer and the normalization layer, please refer to the corresponding part description above, which will not be repeated here.

[0225] In the embodiment of the present application, the model structure other than the hybrid expert layer in the first model can be referred to as a non-hybrid expert layer. Thus, the first model can include a non-hybrid expert layer and a hybrid expert layer. The calculation in which the non-hybrid expert layer participates is referred to as a non-expert calculation. The calculation in which the expert module participates is referred to as an expert calculation. The first expert module can be used to indicate the expert module in the activated state in the first model in the case of the second stage. The second expert module can be used to indicate the expert module in the activated state in the first model in the case of the first stage. The second stage can be used to indicate the decoding stage. The first stage can be used to indicate the pre-filling stage.

[0226] Next, combine Figure 6 The first model shown explains how to deploy the first model on the terminal side device.

[0227] In order to deploy the first model on the end-side device, it is necessary to consider the storage structure of the end-side device. Fig. 7A Provide explanation.

[0228] Fig. 7A A schematic diagram of the storage structure of an end-side device provided in an embodiment of the present application.

[0229] like Fig. 7AAs shown, the storage structure of the end-side device may include a processor memory and a storage device. In addition, the end-side device may also include an accelerator memory. For example, the accelerator memory may include at least one of the following: a graphics processor memory, a neural network processor memory, or a tensor processor memory, etc.

[0230] Data transmission between the processor memory and the accelerator memory can be realized through the processor and the accelerator. The processor can transmit data to the processor memory through the memory bus and the memory controller. The accelerator can transmit data to the accelerator memory through the video memory bus and the video memory controller.

[0231] The processor and the storage device can perform data transmission through the storage interface, the internal bus and the storage controller. The accelerator can perform data transmission with the storage device through the processor. Optionally, the accelerator can perform data transmission directly with the storage device.

[0232] The storage capacity of the storage device may be greater than at least one of the storage capacity (or memory capacity) of the accelerator memory or the storage capacity (or memory capacity) of the processor memory.

[0233] In an embodiment of the present application, as an implementation, the first storage unit may be an accelerator memory. The second storage unit may be a processor memory or a storage device. As another implementation, the first storage unit may be a processor memory. The second storage unit may be a storage device. The first storage unit may be used as a cache for the first expert module and / or the second expert module.

[0234] Since the end-side device has a multi-level storage structure, the embodiment of the present application proposes that the model parameters can be stored in different storage structures to solve the problem of limited memory capacity of the end-side device. For example, the memory capacity can be expanded based on the memory unloading method. Specifically, the model parameters that are not currently involved in the calculation can be unloaded to the second storage unit. In the case where the model parameters need to participate in the calculation, the model parameters are loaded from the second storage unit to the first storage unit, thereby reducing the memory requirements of the first model. However, since the model parameters need to be loaded from the second storage unit to the first storage unit, I / O delays will be generated.

[0235] exist Fig. 7A Based on the following Figure 7B-7D The invention describes expanding memory capacity and I / O delay between a first storage unit and a second storage unit based on a memory offloading method. Figure 7B The first storage unit may be an accelerator memory, and the second storage unit may be a processor memory. Figure 7C The first storage unit may be an accelerator memory, and the second storage unit may be a storage device. Fig.7DThe first storage unit may be a processor memory. The second storage unit may be a storage device. The first storage unit may store a non-hybrid expert layer (i.e., a non-hybrid expert parameter). The second storage unit may store a hybrid expert layer (i.e., a hybrid expert parameter). The non-hybrid expert parameter may be used to indicate a model parameter of the non-hybrid expert layer. The hybrid expert parameter may be used to indicate a model parameter of the hybrid expert layer. Figure 7B-7D This is only an exemplary description and is not intended to limit the embodiments of the present application. For example, the second storage unit may store a non-hybrid expert layer and a hybrid expert layer.

[0236] Figure 7B A schematic diagram of a storage method of a first model provided in an embodiment of the present application.

[0237] like Figure 7B As shown, when it is determined that a certain expert module in the hybrid expert layer is the first expert module, the first expert module (i.e., the first expert parameter) can be loaded from the second storage unit to the first storage unit through the processor and the accelerator. As a result, an I / O delay will be generated between the second storage unit and the first storage unit. The first expert parameter can be used to indicate the model parameter of the first expert module.

[0238] Figure 7C A schematic diagram of another storage method of the first model provided in an embodiment of the present application.

[0239] like Figure 7C As shown, Figure 7B Similarly, the first expert module can be loaded from the second storage unit to the first storage unit through the processor, accelerator and storage device, thereby generating an I / O delay between the second storage unit and the first storage unit.

[0240] Fig.7D A schematic diagram of another storage method of the first model provided in an embodiment of the present application.

[0241] like Fig.7D As shown, Figure 7B Similarly, the first expert module can be loaded from the second storage unit to the first storage unit through the processor and the storage device, thereby generating an I / O delay between the second storage unit and the first storage unit.

[0242] Since there is an I / O delay between the first storage unit and the second storage unit, the reasoning efficiency of the first model is affected. The following describes how to improve the reasoning efficiency in the embodiment of the present application in conjunction with the accompanying drawings.

[0243] Fig. 8A A schematic diagram of the principle of a data processing method provided in an embodiment of the present application.

[0244] like Fig. 8A As shown, in the case where the terminal device is deployed based on the memory offloading method, the data processing process involves non-expert calculation, loading the first expert module from the second storage unit to the first storage unit, and expert calculation. Since there is an I / O delay when loading the first expert module from the second storage unit to the first storage unit, the reasoning efficiency of the first model is affected.

[0245] In order to improve the efficiency of reasoning, it is found that the I / O delay can be covered by preloading and computing in parallel. For example, in the second stage of the first model (i.e., the decoding stage), in the process of executing the expert calculation for the previous word, the second thread can determine the fifth expert module corresponding to the current word from the multiple expert modules included in the first model according to the expert statistical distribution and the first expert module corresponding to the previous word. The fifth expert module corresponding to the current word is preloaded into the first storage unit. The first thread determines the first expert module corresponding to the current word according to the first model. In the case that the fifth expert module corresponding to the first expert module exists in the first storage unit, the fifth expert module corresponding to the first expert module is obtained from the first storage unit. The fifth expert module corresponding to the first expert module is used to perform expert calculation. Therefore, it is expected that the fifth expert module corresponding to the current word can be preloaded and the expert calculation corresponding to the previous word can be performed in parallel, so as to cover the I / O delay and reduce the waiting time of computing resources in the process of loading the first expert module.

[0246] In order to demonstrate the practical effect of the above implementation method, the embodiment of the present application takes the first model as Mixtral MoE-175G, which occupies 19G of processor memory and uses 4 processor cores for processing as an example for experimental verification. MixtralMoE-175G has 175 billion model parameters. The experimental results show that the average time consumed to decode a word is reduced from 26.6s to 21.9s.

[0247] However, it was found that when it was necessary to determine whether the expert module corresponding to the first expert module participated in the calculation, the expert module was still in a loading state, which means that the above implementation method failed to cover the I / O delay caused by loading the expert module. Further research found that this may be caused by at least one of the I / O bottleneck and preloading failure. The I / O bottleneck may refer to the expert calculation being limited by the I / O delay. The preloading duration is longer than the expert calculation duration. Preloading failure may refer to the expert module (i.e., the fifth expert module) stored in the first storage unit based on the preloading method is not the first expert module actually required. Preloading failure can be understood as a cache miss.

[0248] In order to improve the inference efficiency, it is found that this can be achieved by improving the cache hit rate. A cache hit may mean that when a first expert module is needed, the first expert module exists in the first storage unit. Therefore, in the case of a cache hit, the first expert module can be directly obtained from the first storage unit without loading the first expert module from the second storage unit to the first storage unit, thereby reducing I / O latency.

[0249] As to how to improve the cache hit rate, it is found that if as many expert modules as possible can be stored in the first storage unit, this can not only reduce the I / O delay between the second storage unit and the first storage unit, but also improve the cache hit rate. However, due to the limited memory capacity, it is difficult to achieve this by increasing the number of expert modules in the first storage unit.

[0250] It is further found that the use of expert modules has load balancing for the entire data set. Therefore, there may be expert preferences for requests. Therefore, the expert modules that need to be stored in the first storage unit can be determined based on the expert preference strategy of the request. Expert preference can refer to having an expert module corresponding to the request, and the expert modules corresponding to different requests may be different. For example, the expert module corresponding to the first request A may include expert module 1 and expert module 2. The expert module corresponding to the first request B may include expert module 3 and expert module 4. The expert modules corresponding to the first request A and the first request B are different. It can be understood that different requests may refer to the similarity between two requests being less than or equal to the sixth threshold. The sixth threshold can be configured according to actual business needs and is not limited here.

[0251] Therefore, in order to improve the cache hit rate, the embodiment of the present application proposes that an expert preference prediction process based on a request can be adopted to implement it, so that the first storage unit can store the expert module corresponding to the third expert module. The third expert module can be determined from the multiple expert modules included in the first model based on the expert preference strategy of the first request. The third expert module corresponding to the first request may be an expert module that is frequently accessed for processing the first request. The third expert module determined based on the expert preference preloading process of the request can be used in the decoding stage of the first model (i.e., the second stage), and can obtain the first expert module corresponding to the first request from the first storage unit as much as possible, thereby improving the cache hit rate of the decoding stage.

[0252] Next, we need to consider how to combine the request-based expert preference preloading process with the inference phase process of the first model to further reduce the I / O latency of the decoding phase.

[0253] It is further found that according to the above description of the reasoning stage of the large model, in the pre-filling stage (i.e., the first stage), the first model can be used to process at least one input word in parallel. Therefore, although the first model has a high parameter sparsity for each input word, since the expert modules activated by different input words may be different, the parallel processing method reduces the parameter sparsity of the first model as a whole. For ease of understanding, according to the above, the expert module in the pre-filling stage (i.e., the first stage) can be called the second expert module.

[0254] Therefore, in the pre-filling stage, each expert module included in the first model may be an expert module in an activated state, that is, the expert modules included in the first model may be second expert modules. Therefore, it may be necessary to load all the expert modules included in the first model from the second storage unit to the first storage unit, that is, it may be necessary to load all the second expert modules included in the first model from the second storage unit to the first storage unit. In addition, there may be a situation where the expert module is loaded into the first storage unit multiple times. This is because if the memory capacity is limited, if the expert module is activated, the expert module (i.e., the second expert module) can be loaded from the second storage unit to the first storage unit. If the expert module is not activated, the expert module can be removed (or eliminated) from the first storage unit.

[0255] Based on this, in order to further reduce I / O delay, it is found that the pre-filling phase process can be executed in parallel with the request-based expert preference preloading process. This is based on the following considerations: the request-based expert preference preloading process can include a request-based expert preference prediction process and an expert loading process, that is, the request-based expert preference prediction process is executed to obtain the third expert module corresponding to the first request, and then the expert loading process is executed to load the expert module corresponding to the third expert module from the second storage unit to the first storage unit. As a result, the expert loading process will generate I / O delay. According to the above, in the pre-filling phase of the first model, it may be necessary to load all the expert modules of the first model (i.e., the second expert module) from the second storage unit to the first storage unit. Based on this, the expert preference preloading process based on the request can make use of the feature that the pre-filling phase process needs to load the second expert module into the first storage unit, and adjust the expert loading process into an expert retention process, that is, in the process of executing the pre-filling phase process to load the second expert module from the second storage unit to the first storage unit, the expert retention process is executed to retain the second expert module corresponding to the third expert module, so as to hide the I / O delay generated by the expert loading process in the expert preference preloading process based on the request as much as possible. The expert module predicted to be in an activated state in the pre-loading phase can be called the third expert module. The first speed can be greater than the second speed. The first speed can be used to indicate the speed at which the first thread accesses the first storage unit. The second speed can be used to indicate the speed at which the first thread accesses the second storage unit.

[0256] Based on the above, continue to see Fig. 8A The embodiment of the present application proposes to execute the pre-filling phase process and the request-based expert preference preloading process in parallel to improve the cache hit rate of the decoding phase, reduce I / O latency, and thus improve the reasoning efficiency. The first thread can be used to execute the pre-filling phase process, and the second thread can be used to execute the request-based expert preference preloading process. In addition, the first thread can also execute the decoding phase process.

[0257] Combine the following Figure 8B , 8C 8D, how to implement the pre-filling phase process and the request-based expert preference preloading process in parallel is described. For the sake of brevity, the request-based expert preference preloading process is referred to as the preloading process below.

[0258] Figure 8B A schematic diagram of the principle of another data processing method provided in an embodiment of the present application.

[0259] like Figure 8BAs shown, the first thread can be used to execute the first stage process (ie, the pre-filling stage process) and the second stage process (ie, the decoding stage process). The second thread can be used to execute the pre-loading process.

[0260] The first phase process may include sub-processes corresponding to at least one third word. At least one sub-process may be executed in parallel. The sub-process may include non-expert calculation, determining the second expert module, loading the second expert module from the second storage unit to the first storage unit and expert calculation, etc. The third word may be determined according to the first request using the first thread. The third word may be used to indicate an input word.

[0261] The second stage process may include non-expert calculation, determining the first expert module, obtaining the second expert module and expert calculation corresponding to the first expert module from the first storage unit, etc.

[0262] The preloading process may include request-based expert preference prediction and retaining the second expert module corresponding to the third expert module in the first storage unit. The request-based expert preference prediction may be implemented as follows. As an implementation, multiple third expert modules corresponding to the first request may be determined based on the first request.

[0263] The first thread may load the second expert module from the second storage unit to the first storage unit. The second expert module may be determined by the first thread according to the first request using the first model. When the first thread executes the first stage process, the second thread may retain the second expert module corresponding to the third expert module in the first storage unit. Thus, the preloading process executed by the second thread and the first stage process executed by the first thread are executed in parallel, and the second expert module corresponding to the third expert module in the first storage unit is retained, thereby hiding the I / O delay of the preloading process.

[0264] Since the third expert module is an expert module that may be accessed when the first thread is used to execute the second stage process, the first thread can obtain the first expert module from the multiple second expert modules in the first storage unit when the first model is used to execute the second stage process, and then use the first expert module to perform expert calculations. Since the first thread does not need to load the first expert module corresponding to the first expert module from the second storage unit to the first storage unit when executing the second stage using the first model, the I / O delay between the first storage unit and the second storage unit is reduced, thereby improving the reasoning efficiency of the first model.

[0265] Combine the following Figure 8C and Fig.8D , how to make predictions based on the expert preferences of requests is explained.

[0266] Figure 8C A schematic diagram of the principle of a request-based expert preference prediction method provided in an embodiment of the present application.

[0267] like Figure 8C As shown, the second model can be used to obtain second scores corresponding to the plurality of expert modules according to the first request. According to the second scores corresponding to the plurality of expert modules, a plurality of third expert modules are determined. The second scores can be used to indicate the predicted activation scores of the expert modules. The predicted activation scores can be used to indicate the probability of the expert modules being accessed.

[0268] The second model may be obtained by training using at least one second request and multiple first scores corresponding to each of the at least one second request. At least one second request may be used as a sample. For the description of the second request, please refer to the description of the request above, which will not be repeated here. In addition, the embodiment of the present application does not limit the acquisition method of the second request. Multiple first scores corresponding to each of the at least one second request may be used as labels. The multiple first scores corresponding to the second request may include first scores corresponding to each of the multiple expert modules. The first score may be used to indicate the activation score of the expert module. The activation score is the actual activation score. The second model may be obtained by offline training. The second model may be trained using at least one of the end-side device or the non-end-side device. In the case where the second model may be trained using the end-side device, the end-side device for training the second model may be the same as or different from the end-side device for applying the second model, which is not limited in the embodiment of the present application. The model structure of the second model may be configured according to actual business needs and is not limited here. For example, when the second model can meet the prediction accuracy, the model parameters of the second model are as small as possible to reduce the resource consumption of the end-side model.

[0269] Regarding how to train the second model using at least one second request and a plurality of first scores respectively corresponding to the at least one second request, it can be implemented in the following manner.

[0270] As an implementation method, the second request can be input into the second model to obtain a second score corresponding to each of the multiple expert modules. Based on the loss function, a loss function value is obtained according to the first score and the second score corresponding to each of the multiple expert modules. The model parameters of the second model are adjusted according to the loss function value to obtain a trained second model. The loss function can be configured according to actual business needs and is not limited here. For example, the loss function may include a cross entropy loss function.

[0271] Regarding how to obtain the loss function value based on the loss function according to the first scores and the second scores corresponding to each of the multiple expert modules, it can be achieved in the following manner.

[0272] As an implementation method, the first score and the second score corresponding to the expert module can be input into the loss function to obtain the loss function value corresponding to the expert module. The loss function value is obtained according to the loss function values ​​corresponding to each of the multiple expert modules.

[0273] The following describes how to determine a plurality of first scores corresponding to each of the at least one second request.

[0274] The first model may include at least one first layer of cascade. The first layer may include a gating module and a plurality of expert modules. The reasoning phase of the first model may include a pre-filling phase and a decoding phase. For any second request in at least one second request, in the pre-filling phase, the second model may be used to obtain a second word corresponding to the second request (i.e., the first output word) according to at least one third word corresponding to the second request (i.e., the input word). In the decoding phase, the second model and an autoregressive method may be used to obtain first words (i.e., output words) one by one according to the second word corresponding to the second request and at least one third word. The second word may also be referred to as the first word.

[0275] In the case of performing the pre-filling phase process using the second model, a second score corresponding to each of the plurality of expert modules corresponding to each of the at least one first layer may be determined for the third word. In the case of performing the decoding phase process using the second model, a second score corresponding to each of the plurality of expert modules corresponding to each of the at least one first layer may be determined for the first word.

[0276] Based on this, the multiple expert modules corresponding to the second request may include multiple expert modules corresponding to at least one third word. Optionally, the multiple expert modules corresponding to the second request may include multiple expert modules corresponding to at least one third word and multiple expert modules corresponding to at least one first word. Optionally, the multiple expert modules corresponding to the second request may include multiple expert modules corresponding to at least one first word.

[0277] For any third word-gram in at least one third word-gram, for any expert module in the multiple expert modules corresponding to the third word-gram, a first score corresponding to the expert module can be obtained. For any first word-gram in at least one first word-gram, for any third expert module in the multiple expert modules corresponding to the first word-gram, a first score corresponding to the expert module can be obtained. Thus, the first scores corresponding to each of the multiple expert modules corresponding to the second request can be determined.

[0278] In order to facilitate understanding of how to determine the first scores corresponding to the multiple expert modules corresponding to the third word or the first scores corresponding to the multiple expert modules corresponding to the first word, the following takes a third word as an example. Fig.8D Provide explanation.

[0279] Fig.8D A schematic diagram of determining first scores corresponding to a plurality of expert modules corresponding to a third word-gram provided in an embodiment of the present application.

[0280] like Fig.8D As shown, the multiple expert modules corresponding to the third word-gram may include multiple expert modules corresponding to at least one first layer respectively.

[0281] For any first layer of at least the first layer included in the first model, the gating module included in the first layer can be used to obtain first scores corresponding to the plurality of expert modules included in the first layer according to the third word unit. Thus, the first scores corresponding to the plurality of expert modules corresponding to the third word unit can be determined.

[0282] In order to prove the actual effect of the parallel execution of the pre-filling phase process and the pre-loading process proposed in the embodiment of the present application, the following experimental verification is carried out. When the first model can be DeepSeek MoE-32G, and the first storage unit is configured to store approximately 10% (ie 3.1G) of the model parameters of the first model, the average cache hit rate can be increased from 0.08 to 0.25. The end-to-end decoding overhead is reduced by an average of about 25%. When the first model is Mixtral MoE-175G and 4 processor cores are used for calculation, if the processor memory occupied is 19G, the average time consumed to decode a word element is reduced from 26.6s to 16.2s. If the processor memory occupied is 10G, the average time consumed to decode a word element is reduced from 29.1s to 19.12s.

[0283] The above illustrates the inventive concept proposed in an embodiment of the present application of executing the pre-filling phase process executed by the first thread in parallel with the pre-loading process executed by the second thread to improve the cache hit rate, reduce I / O latency, and thereby improve the reasoning efficiency of the first model.

[0284] It is further found that when the first thread uses the first model to execute the decoding phase process, there may be a cache miss, that is, the first storage unit does not have the second expert module corresponding to the first expert module. As a result, the first thread will not be able to obtain the second expert module corresponding to the first expert module from the first storage unit. In this case, the first thread needs to first load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, and then obtain the expert module corresponding to the first expert module from the first storage unit, thereby generating an I / O delay between the second storage unit and the first storage unit.

[0285] In order to further reduce I / O delay, it is found that if the first storage unit has a second expert module corresponding to the fourth expert module, and the fourth expert module can be used to replace the first expert module, the I / O delay can be reduced. Therefore, the embodiment of the present application proposes a processing method based on a cache priority strategy.

[0286] Next, it is necessary to consider under what conditions the fourth expert module can be used to replace the first expert module. It is found that the conditions that the fourth expert module used to replace the first expert module needs to meet can be determined from the perspective of the weight of the expert module. For example, the first expert module can have a first weight. The fourth expert module can have a second weight. The first weight can be used to indicate the activation probability of the first expert module. The second weight can be used to indicate the activation probability of the fourth expert module. In order to enable the fourth expert module to replace the first expert module, the absolute value of the difference between the first weight and the second weight can be made as small as possible. Therefore, the embodiment of the present application proposes that the absolute value of the difference between the first weight and the second weight can be less than or equal to the first threshold. The first threshold can be configured according to actual business needs and is not limited here. For example, the first threshold can be 0.3.

[0287] Based on this, when the first thread determines the first expert module corresponding to the first request from the multiple expert modules included in the first model according to the first word corresponding to the first request, if it is determined according to the first weight and the first threshold that there is a fourth expert module corresponding to the first expert module among the multiple expert modules included in the first model, it means that when there is no second expert module corresponding to the first expert module in the first storage unit, the fourth expert module can be considered to replace the first expert module. If there is a second expert module corresponding to the fourth expert module in the first storage unit, the first expert module can be replaced by the fourth expert module.

[0288] The following describes how the second expert module corresponding to the fourth expert module is stored in the first storage unit.

[0289] The first thread and the second thread may also process a third request while processing the first request, and the third request may be different from the first request. Therefore, the fourth expert module and the first expert module may correspond to the same request. For example, the fourth expert module and the first expert module correspond to the first request. Therefore, the fourth expert module may be the second expert module corresponding to the first request stored in the first storage unit. In addition, the fourth expert module and the first expert module may correspond to different requests. For example, the fourth expert module corresponds to the third request, and the first expert module corresponds to the first request. Therefore, the fourth expert module may be the second expert module corresponding to the third request stored in the first storage unit.

[0290] For ease of understanding, the following will combine Fig.9A and Fig. 9B It indicates that the first thread replaces the first expert module with the fourth expert module when the first storage unit does not contain the second expert module corresponding to the first expert module and the first storage unit contains the second expert module corresponding to the fourth expert module. Fig.9A This is the case where the fourth expert module and the first expert module correspond to the same request. Fig. 9B This is the case where the fourth expert module and the first expert module correspond to different requests.

[0291] Fig.9A A schematic diagram of replacing a first expert module with a fourth expert module is provided in an embodiment of the present application.

[0292] like Fig.9A As shown, the first model may include a first layer 900. The first layer 900 may include a gating module 910 and four expert modules. The four expert modules may include an expert module 921, an expert module 922, an expert module 923, and an expert module 924.

[0293] The first thread can use the gating module 910 to determine expert module 921 and expert module 923 as the first expert module from expert module 921, expert module 922, expert module 923 and expert module 924 according to the first word corresponding to the first request. The absolute value of the difference between the third weight corresponding to expert module 922 and the third weight of expert module 923 is less than or equal to the first threshold. Therefore, expert module 922 can be used as the fourth expert module corresponding to expert module 923 (i.e., the first expert module). The third weight corresponding to expert module 922 can be called the second weight. The third weights corresponding to expert module 921 and expert module 923 can both be called the first weight.

[0294] The first thread determines that the first storage unit has a second expert module corresponding to the expert module 921 (ie, the first expert module), and thus, the second expert module corresponding to the first expert module can be obtained from the first storage unit.

[0295] The first thread determines that the first storage unit does not have a second expert module corresponding to the expert module 923 (i.e., the first expert module), but the first storage unit has a second expert module corresponding to the fourth expert module (i.e., expert module 922), and the expert module 922 is the fourth expert module corresponding to the expert module 923 (i.e., the first expert module). Therefore, the first expert module (i.e., expert module 923) can be replaced by the fourth expert module (i.e., expert module 922). Then, the first thread can obtain the second expert module corresponding to the fourth expert module from the first storage unit, thereby improving the cache hit rate. And since there is no need to first load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, and then obtain the expert module corresponding to the first expert module from the first storage unit, the I / O delay between the second storage unit and the first storage unit is reduced.

[0296] Based on this, the first thread may utilize the first expert module (ie, expert module 921 ) and the fourth expert module (ie, expert module 922 ) to process the first word.

[0297] Fig. 9B Another schematic diagram of using a fourth expert module to replace a first expert module is provided in an embodiment of the present application.

[0298] like Fig. 9B As shown, Fig.9A The difference is that the fourth expert module (i.e., expert module 922) corresponding to the first expert module (i.e., expert module 923) is the second expert module corresponding to the third request stored in the first storage unit. The third request is different from the first request. The other parts are the same as Fig.9A Similar, no further elaboration is required here.

[0299] In order to prove the actual effect of the processing method based on the cache priority strategy proposed in the embodiment of the present application, the following experimental verification is carried out. When the first model can be DeepSeek MoE-32G, and the first storage unit is configured to store about 10% (i.e. 3.1G) of the model parameters of the first model, the average cache hit rate can be increased from 0.25 to 0.60 based on the parallel execution of the above-mentioned pre-filling phase process and the pre-loading process. The end-to-end decoding overhead is reduced by an average of about 41.6%.

[0300] It is further found that, in the case of a cache miss, it is also possible to consider using a third thread to obtain an expert module corresponding to the first expert module from the second storage unit, and use the expert module corresponding to the first expert module to process the first word. The third speed may be greater than the fourth speed. The third speed may be used to indicate the speed at which the third thread accesses the second storage unit. The fourth speed may be used to indicate the speed at which the third thread accesses the first storage unit. For example, the third thread may be a processor thread. The first thread may be an accelerator thread. This is because it is found that the average time consumed by using the third thread to process the first word is less than the average time consumed by the first thread to load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtain the expert module corresponding to the first expert module from the first storage unit, and use the expert module corresponding to the first expert module to process the first word.

[0301] In order to understand how to use the third thread to process the first word, the following Fig.10 Provide explanation.

[0302] Fig.10 A schematic diagram of using a third thread to process a first word according to an embodiment of the present application.

[0303] like Fig.10 As shown, the first model may include a first layer 1000. The first layer 1000 may include a gating module 1010 and four expert modules. The four expert modules may include an expert module 1021, an expert module 1022, an expert module 1023, and an expert module 1024.

[0304] The first thread may utilize gating module 1010 to determine expert module 1021 and expert module 1023 as the first expert module from among expert module 1021, expert module 1022, expert module 1023, and expert module 1024 according to the first word corresponding to the first request.

[0305] The first thread determines that the first storage unit has a second expert module corresponding to the expert module 1021 (ie, the first expert module), and thus, the second expert module corresponding to the first expert module can be obtained from the first storage unit.

[0306] The first thread determines that the first storage unit does not contain a second expert module corresponding to the expert module 1023 (ie, the first expert module). The third thread can obtain an expert module corresponding to the expert module 1023 (ie, the first expert module) from the second storage unit.

[0307] Based on this, the first thread can use the second expert module corresponding to the first expert module (ie, expert module 1021) to process the first word-gram. The third thread can use the expert module corresponding to the first expert module (ie, expert module 1023) to process the first word-gram.

[0308] Since the memory capacity of the first storage unit is limited, in the process of executing the data processing method based on the first thread and the second thread, there may be a situation where the first storage unit meets the first condition, thereby affecting the cache hit rate and resource utilization. The first condition can be used to trigger cache elimination. As an implementation, the first condition can be used to indicate that the first storage unit is full.

[0309] In order to improve the cache hit rate and resource utilization, the embodiment of the present application proposes that if the first storage unit meets the first condition, the fourth thread can be used to execute the cache elimination process based on the cache elimination strategy, thereby eliminating the second expert module among the multiple second expert modules stored in the first storage unit, thereby improving the cache hit rate and resource utilization.

[0310] For example, the fourth thread may eliminate the second expert module from the plurality of second expert modules stored in the first storage unit based on the cache elimination strategy when the first storage unit meets the first condition. The fourth thread may be executed in parallel with the first thread and the second thread.

[0311] Based on the above, the embodiment of the present application builds a first model based on MoE, completes the scheduling of model parameters from the second storage unit to the first storage unit, and can configure a predetermined number of expert modules to be stored in the first storage unit. In addition, pipeline parallel processing of calculating and storing the model parameters of the expert modules in the first storage unit is realized. The cache hit rate of the first storage unit is improved by using the expert preference strategy based on the request. In addition, the cache hit rate can be further improved by using the cache priority strategy.

[0312] The above describes the inventive concept of the embodiment of the present application. The data processing method provided by the embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0313] First, combine Fig.11 and Fig.12 , the end-side device to which the data processing method is applicable is described. Fig.11 and Fig.12 Taking the end-side device shown in the figure as an example, in conjunction with the accompanying drawings (for example, Fig.13 , Fig.14A , Fig. 14B , Fig. 14C , Fig.14D , Fig.14E , Fig.15 , Fig.16 , Fig.17 , Fig.18 ) and application scenarios, the data processing method provided in the embodiment of the present application is specifically described.

[0314] It should be noted that the embodiments of the present application can be implemented independently or in conjunction with each other, and the same or similar concepts or processes will not be repeated in some embodiments.

[0315] Fig.11 A schematic diagram of the structure of the terminal device provided in an embodiment of the present application.

[0316] The terminal device 1100 may include a processor 1110, an external memory interface 1120, an internal memory 1121, a universal serial bus (USB) interface 1130, a charging management module 1140, a power management module 1141, a battery 1142, an antenna 1, an antenna 2, a mobile communication module 1150, a wireless communication module 1160, an audio module 1170, a speaker 1170A, a receiver 1170B, a microphone 1170C, an earphone interface 1170D, a sensor module 1180, a button 1190, a motor 1191, an indicator 1192, a camera 1193, a display screen 1194, and a subscriber identification module (SIM) card interface 1195, etc. Among them, the sensor module 1180 may include a pressure sensor 1180A, a gyroscope sensor 1180B, an air pressure sensor 1180C, a magnetic sensor 1180D, an acceleration sensor 1180E, a distance sensor 1180F, a proximity light sensor 1180G, a fingerprint sensor 1180H, a temperature sensor 1180J, a touch sensor 1180K, an ambient light sensor 1180L, a bone conduction sensor 1180M, etc.

[0317] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 1100. In other embodiments of the present application, the terminal device 1100 may include more or fewer components than shown in the figure, or combine certain components, or split certain components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0318] Processor 1110 may include one or more processing units. For example, processor 1110 may include an application processor (AP). The application processor may include a central processing unit, a hardware accelerator, a memory controller, a video codec, and a baseband processor (BP). The hardware accelerator may include at least one of the following: a graphics processor, a neural network processor, a tensor processor, an image signal processor (ISP), or a digital signal processor (DSP), etc. Among them, different processing units may be independent devices or integrated in one or more processors. For example, the application processor may be a system on chip. In an embodiment of the present application, a hardware accelerator may be referred to as an accelerator.

[0319] The processor 1110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 1110 is a cache memory. The memory may store instructions or data that the processor 1110 has just used or cyclically used. If the processor 1110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 1110, and thus improves the efficiency of the system.

[0320] In some instances, the memory of the embodiment of the present application may store instructions and data for implementing the data processing method described in the embodiment of the present application, and when the processor is running, the data processing method described in the embodiment of the present application may be implemented.

[0321] The internal memory 1121 may be used to store computer executable program codes, which include instructions. The internal memory 1121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system or an application required for at least one function (for example, an AI application, etc.). The data storage area may store data created during the use of the terminal device 700, etc. In addition, the internal memory 1121 may include a high-speed random access memory, and may also include a non-volatile memory, for example, at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 1110 executes various functional applications and data processing of the terminal device 700 by running instructions stored in the internal memory 1121 and / or instructions stored in a memory provided in the processor 1110. In an embodiment of the present application, the internal memory 1121 may be used to implement instructions and data of the data processing method described in the embodiment of the present application.

[0322] The software system of the terminal device 1100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 1100.

[0323] Fig.12 A software structure block diagram of the terminal device provided in an embodiment of the present application.

[0324] The layered architecture can divide the software into multiple layers, each with a clear role and division of labor. The layers can communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer (i.e., framework layer), the hardware adaptation layer (HAL), the driver layer (i.e., driver layer), and the hardware layer.

[0325] The application layer may include an application package. In an embodiment of the present application, the application package may include an AI application. As an implementation method, when the end-side device runs an AI application, the AI ​​application may execute the data processing method of the embodiment of the present application.

[0326] The application framework layer provides an application programming interface (API) and a programming framework for applications in the application layer. The application framework layer may include predefined functions. In an embodiment of the present application, the application framework layer may include a neural network API (NNAPI). The neural network API is a system-level application programming interface. The neural network API may be a framework provided by Android for running a terminal model on a terminal device. The neural network API may support at least one of loading a terminal model obtained using a machine learning framework, compiling the terminal model into a hardware executable format, or determining hardware for performing reasoning tasks. The machine learning framework may be a framework that supports running a machine learning model on a terminal device. For example, the hardware may include an application processor. The application processor may include a central processing unit and a hardware accelerator. The hardware accelerator may include at least one of the following: a graphics processor, a neural network processor, or a tensor processor.

[0327] The hardware abstraction layer is an interface layer located between the application framework layer and the driver layer, which can be an encapsulation of the hardware driver and provide a unified interface for the call of the upper layer application. In an embodiment of the present application, the hardware abstraction layer may include an AI hardware abstraction layer. The AI ​​hardware abstraction layer can be used to transmit requests for AI applications to the hardware. The AI ​​hardware abstraction layer can also be used to compile the end-side model into a hardware executable format or assign reasoning tasks to the hardware. The AI ​​hardware abstraction layer may include an AI hardware abstraction layer interface. The AI ​​hardware abstraction layer interface can be used to transmit requests for AI applications to the hardware of the driver layer. The AI ​​hardware abstraction layer interface may include a first interface, a second interface, and a third interface. The first interface may be used to provide basic information about the hardware. For example, based on the information, at least one of the following may be included: operation type or performance indicator. The second interface may be used to load and compile the end-side model. The third interface may be used to perform reasoning tasks.

[0328] The neural network application programming interface can call the AI ​​hardware abstraction layer interface to compile the end-side model into a hardware executable format. The neural network application programming interface can determine the hardware used to perform the reasoning task based on the basic information of the hardware provided by the AI ​​hardware abstraction layer. For example, the neural network application programming interface can determine that the hardware used to perform the reasoning task is the central processing unit when there is no hardware accelerator on the end-side device. Optionally, the neural network application program interface can determine that the hardware used to perform the reasoning task is the hardware accelerator when there is a hardware accelerator on the end-side device. Optionally, the neural network application program interface can determine that the hardware used to perform the predetermined operation is the central processing unit when there is a hardware accelerator and a predetermined operation. The predetermined operation may refer to an operation that needs to be performed by the central processing unit. The AI ​​hardware abstraction layer can transmit reasoning tasks to the hardware and manage reasoning tasks.

[0329] The driver layer may be a layer between hardware and software, which may be used to drive the hardware. In an embodiment of the present application, the driver layer may include an Android kernel. The Android kernel may be used to manage hardware and provide system services to an upper layer (e.g., a hardware abstraction layer or an application framework layer). System services may include at least one of the following: process scheduling, memory management, or a file system, etc. Hardware drivers may be written for a central processing unit and hardware accelerators, etc., to support the end-side model. The hardware driver is integrated into the Android kernel.

[0330] The application layer can indirectly use the services provided by the Android kernel through the application framework layer. The AI ​​hardware abstraction layer can access the hardware through the Android kernel. The application framework layer can access the Android kernel through system calls.

[0331] The hardware layer may include a central processing unit, a graphics processing unit, a neural network processor, and a tensor processor, etc., for performing inference tasks.

[0332] It should be noted that the embodiments of the present application are only illustrated by the Android system. In other operating systems, if the functions implemented by each functional module are similar to those in the embodiments of the present application, the data processing method described in the embodiments of the present application can also be implemented.

[0333] Combine the following Fig.13 , Fig.14A , Fig. 14B , Fig. 14C , Fig.14D , Fig.14E , Fig.15 , Fig.16 and Fig.17 The data processing method based on the first thread, the second thread, the third thread, the fourth thread, the first storage unit, the second storage unit, the first model and the second model provided in the embodiment of the present application is described. Fig.13 , Fig.15 , Fig.16 and Fig.17 The difference lies in that: different implementation methods are provided for the situation where the first storage unit does not have a second expert module corresponding to the first expert module. Figure 14A-14E Shown for Fig.13 Specific examples.

[0334] For ease of understanding, the first model, the first storage unit, the second storage unit and the third thread are supplementarily explained below in combination with the above, and the first set and the second set involved below are explained.

[0335] The reasoning phase of the first model may include a first phase and a second phase. The first phase may also be referred to as a pre-filling phase. The second phase may also be referred to as a decoding phase. According to the above, the pre-filling phase may refer to a phase in which the first round of processing is performed. The decoding phase may refer to a phase in which the second to Pth rounds of processing are performed.

[0336] The first model may include multiple first layers in cascade. The first layer may include a mixed expert layer. The mixed expert layer may include a gating module and multiple expert modules. The gating module may be used to determine an expert module in an activated state from multiple expert modules. The expert module in an activated state may be used to perform nonlinear transformation and feature extraction on the input. For example, the first model may include R first layers in cascade, i.e., the 1st first layer, ..., the rth first layer, ..., the Rth first layer. The rth first layer may include the rth mixed expert layer. The rth mixed expert layer may include S expert modules, i.e., the r_1th expert module, ..., the r_sth expert module, ..., the r_Sth expert module. R may be an integer greater than or equal to 1. r may be an integer greater than or equal to 1 and less than or equal to R. S may be an integer greater than 1. s may be an integer greater than or equal to 1 and less than or equal to S.

[0337] For example, the first model could be Figure 6 The model shown in Figure 1. The first layer of the rth layer can refer to Figure 6 The first layer 600_r in the r-th mixed expert layer can be Figure 6 The r_s expert module may refer to Figure 6 Medium expert module 612_r_s.

[0338] It should be noted that the first layer currently being used may be referred to as the current layer. For example, the rth first layer may be used to indicate the current layer. In the first stage (i.e., the pre-filling stage), the expert module in the activated state may be referred to as the second expert module. In the second stage (i.e., the decoding stage), the expert module in the activated state may be referred to as the first expert module. The first expert module may have a first identifier. The expert module may have a second identifier. The second expert module may have a third identifier. The first identifier may be used to indicate the first expert module. The second identifier may be used to indicate the expert module. The third identifier may be used to indicate the second expert module.

[0339] The second model may be used to determine the third expert module that needs to be preloaded into the first storage unit. The second model may be trained using the method described above. In the preloading stage, the expert module predicted to be in an activated state may be referred to as the third expert module. The third expert module may have a fourth identifier. The fourth identifier may be used to indicate the third expert module.

[0340] The first storage unit can be used to store the second expert module corresponding to the third expert module. The second expert module corresponding to the third expert module can be understood as the second expert module is the third expert module. The first storage unit storing the second expert module can be understood as the first storage unit storing the model parameters of the second expert module. The model parameters of the second expert module can be associated with the third identifier. The number of second expert modules stored in the first storage unit can be configured according to actual business needs and is not limited here.

[0341] The second storage unit can be used to store the expert module included in the first model. The second storage unit storing the expert module can be understood as the second storage unit storing the model parameters of the expert module.

[0342] The second set may include multiple second expert modules. The second set may be obtained by the first thread loading the expert module corresponding to the second expert module from the second storage unit to the first storage unit.

[0343] The first set may include the second expert module corresponding to the third expert module. The first set may be obtained by the second thread retaining the second expert module corresponding to the third expert module in the second set.

[0344] The third thread can be used for the first thread to obtain the expert module corresponding to the first expert module from the second storage unit and execute the second stage process using the expert module corresponding to the first expert module when it is determined that the second expert module corresponding to the first expert module does not exist in the first storage unit, or when it is determined that both the second expert module corresponding to the first expert module and the second expert module corresponding to the fourth expert module do not exist in the first storage unit.

[0345] The data processing method based on the first thread, the second thread, the third thread, the fourth thread, the first storage unit, the second storage unit, the first model and the second model provided in an embodiment of the present application is described below in conjunction with the accompanying drawings.

[0346] Fig.13 A flowchart of a data processing method provided in an embodiment of the present application. The method can be applied to a first thread and a second thread.

[0347] like Fig.13 As shown, the method includes S1301-S1316. The first thread can use the first model to execute the first stage process (i.e., the pre-filling stage process) and the second stage process (i.e., the decoding stage process). The second thread can use the second model to execute the preloading process. The first thread using the first model to execute the first stage process can be executed in parallel with the second thread using the second model to execute the preloading process.

[0348] The pre-filling phase process (ie, the first phase process) may include S1301 - S1303. The decoding phase process (ie, the second phase process) may include S1306 - S1316. The pre-loading process may include S1304 - S1305.

[0349] According to the above, the pre-filling stage may refer to the stage of performing the first round of processing. The decoding stage may refer to the stage of performing the second to Pth rounds of processing. Thus, Fig.13 The decoding phase process may refer to the decoding phase process corresponding to the pth round. p may be an integer greater than or equal to 2 and less than or equal to P. P may be an integer greater than 1. The first thread may repeatedly execute S1306-S1315 based on an autoregressive method for rounds 2 to P until an inference result (i.e., output result) is obtained. The input corresponding to the pth round may include at least one third word element corresponding to the first request, and the first word element corresponding to each of rounds 1 to p-1. The output corresponding to the pth round may include the first word element corresponding to the pth round. The third word element may be used to indicate an input word element. The first word element may be used to indicate an output word element. The first word element corresponding to the 1st round may also be referred to as a second word element.

[0350] The first thread may execute the first phase process (i.e., the pre-filling phase process) using the first model in parallel with the second thread executing the pre-loading process using the second model. The pre-filling phase process, the pre-loading process, and the decoding phase process are described below.

[0351] How the first thread uses the first model to execute the first phase process (ie, the pre-filling phase process) can be implemented through the following S1301 - S1303.

[0352] In S1301, the first thread determines, according to at least one third word-gram corresponding to the first request, at least one second expert module corresponding to each of the at least one third word-grams from a plurality of expert modules included in the first model to obtain a second set.

[0353] According to an embodiment of the present application, the first request may be used to indicate a task to be processed by the first model. For the description of the first request, please refer to the description of the request above, which will not be repeated here.

[0354] The third word-element may be used to indicate an input word-element. At least one third word-element may be obtained by the first thread tokenizing the first request.

[0355] For example, the first request may be “in which city is FD University located”. The at least one third word element corresponding to the first request may include a third word element [“FD”], a third word element [“university”], a third word element [“located in”], a third word element [“which city”], and a third word element [“city”].

[0356] According to an embodiment of the present application, as described above, the multiple expert modules included in the first model may include multiple expert modules corresponding to at least one first layer, that is, the first model may include at least one first layer. Each first layer may include multiple expert modules. The at least one second expert module corresponding to the third word may include at least one second expert module corresponding to at least one first layer, that is, each first layer may have at least one second expert module corresponding to the first layer.

[0357] According to an embodiment of the present application, the second set may include at least one second expert module corresponding to each of the at least one third word-grams.

[0358] How to determine at least one second expert module corresponding to at least one third word-gram can be implemented in the following manner.

[0359] As an implementation, the first thread may determine, based on the at least one third word element corresponding to the first request, at least one second expert module corresponding to each of the at least one third word elements from a plurality of expert modules included in the first model and corresponding to each of the at least one first layer. The steps of determining the at least one second expert module corresponding to each of the at least one third word elements may be performed in parallel.

[0360] How the first thread determines at least one second expert module corresponding to at least one third word from a plurality of expert modules corresponding to at least one first layer can be implemented in the following manner.

[0361] As an implementation, the first layer may include a gating module. Thus, the first thread may use the gating module corresponding to each of the at least one first layer to determine, according to the at least one third word corresponding to the first request, at least one second expert module corresponding to each of the at least one third word from the multiple expert modules corresponding to each of the at least one first layer. For example, for any third word in the at least one third word, any first layer in the at least one first layer, the first layer is taken as the current layer, and the first thread may use the gating module corresponding to the current layer to determine, according to the third word, at least one second expert module corresponding to the third word and the current layer from the multiple expert modules corresponding to the current layer.

[0362] How to determine at least one second expert module corresponding to the third word-gram and the current layer can be implemented in the following manner.

[0363] As an implementation, the first thread may use the gating module corresponding to the current layer to obtain the fourth weights corresponding to the multiple expert modules corresponding to the current layer according to the third word unit. The first thread may determine at least one second expert module corresponding to the third word unit from the multiple expert modules corresponding to the current layer according to the fourth weights corresponding to the multiple expert modules corresponding to the current layer.

[0364] How the first thread determines at least one second expert module corresponding to the third word from the multiple expert modules corresponding to the current layer according to the fourth weights corresponding to the multiple expert modules corresponding to the current layer can be implemented in the following manner.

[0365] The first method is based on the Top-T method. For example, the first thread can determine T fourth weights from the fourth weights corresponding to each of the multiple expert modules corresponding to the current layer. The expert modules corresponding to the T fourth weights are determined as the second expert modules corresponding to the third word and the current layer. When the larger the fourth weight is, the greater the probability that the expert module is activated is, the T fourth weights can be the maximum T fourth weights. When the smaller the fourth weight is, the greater the probability that the expert module is activated is, the T fourth weights can be the minimum T fourth weights. T can be an integer greater than or equal to 1 and less than or equal to S. T can be configured according to actual business needs and is not limited here. For example, S=16. T=2.

[0366] The second method is based on the threshold method. For example, when the fourth weight is larger, the probability of the expert module being activated is larger, and the first thread can determine the fourth weight greater than or equal to the sixth threshold from the fourth weights corresponding to the multiple expert modules corresponding to the current layer. The expert module corresponding to the fourth weight greater than or equal to the seventh threshold is determined as the second expert module corresponding to the third word and the current layer. When the fourth weight is smaller, the probability of the expert module being activated is larger, and the first thread can determine the fourth weight less than or equal to the eighth threshold from the fourth weights corresponding to the multiple expert modules corresponding to the current layer. The expert module corresponding to the fourth weight less than or equal to the eighth threshold is determined as the second expert module corresponding to the third word and the current layer. The seventh threshold may be greater than the eighth threshold. The seventh threshold and the eighth threshold can be configured according to actual business needs and are not limited here.

[0367] According to the above, the first model may include R first layers in cascade, i.e., the 1st first layer, ..., the rth first layer, ..., the Rth first layer. The rth first layer may include the rth mixed expert layer. The rth mixed expert layer may include the rth gating module and S rth expert modules. The S rth expert modules may include the r_1th expert module, ..., the r_sth expert module, ..., the r_Sth expert module. For the description of the first model, please refer to the corresponding part above, which will not be repeated here. The at least one second expert module corresponding to the third word may include at least one second expert module corresponding to each of the R first layers. The current layer described above may refer to the rth first layer. The gating module corresponding to the current layer may refer to the rth gating module. The expert module corresponding to the current layer may refer to the r_sth expert module.

[0368] In the case of r=1, the first thread can input the 1st first feature into the 1st gating module to obtain the fourth weight corresponding to each of the S 1st expert modules, that is, the fourth weight corresponding to each of the 1_1th expert module, ..., the 1_sth expert module, ..., and the 1_Sth expert module. The fourth weight corresponding to the 1_sth expert module can be used to indicate the activation probability of the 1_sth expert module. The first thread can determine at least one second expert module corresponding to the third word element from the S 1st expert modules based on the fourth weights corresponding to each of the S 1st expert modules. The 1st first feature can be obtained by the first thread based on the third word element using the 1st first layer.

[0369] In the case of 1<r≤R, the first thread can input the r first feature into the rth gating module to obtain the fourth weight corresponding to each of the S rth expert modules, that is, the fourth weight corresponding to each of the r_1th expert module, ..., r_sth expert module, ..., r_Sth expert module. The fourth weight corresponding to the r_sth expert module can be used to indicate the activation probability of the r_sth expert module. The first thread can determine at least one second expert module corresponding to the third word unit from the S rth expert modules based on the fourth weights corresponding to each of the S rth expert modules. The rth first feature can be obtained by the first thread based on the third word unit using the first layer.

[0370] In S1302, the first thread loads at least one second expert module corresponding to each of the at least one third word-grams from the second storage unit to the first storage unit to obtain a second set.

[0371] According to an embodiment of the present application, the second set may include at least one second expert module corresponding to at least one third word-gram, and the at least one second expert module corresponding to the third word-gram may include at least one second expert module corresponding to at least one first layer.

[0372] For any one of the at least one third word-gram and any one of the at least one first layer, with the first layer as the current layer, the first thread may load at least one second expert module corresponding to the third word-gram and the current layer from the second storage unit to the first storage unit. The loading steps for at least one second expert module corresponding to each of the at least one third word-gram may be performed in parallel.

[0373] At 1303 , the first thread uses the second set to obtain a second word-gram corresponding to the first request according to at least one third word-gram.

[0374] According to an embodiment of the present application, the first thread may use at least one second expert module included in the first model and corresponding to each of the at least one third word-element to execute the first stage process to obtain a second word-element corresponding to the first request. The second word-element may be used to indicate the first output word-element corresponding to the first request. For example, the second word-element may be Figure 3 The first output word ["FD"].

[0375] How the second thread uses the second model to execute the preloading process can be implemented through the following S1304-S1305.

[0376] In S1304, the second thread uses the second model to obtain multiple third expert modules corresponding to the first request according to the first request.

[0377] According to an embodiment of the present application, the second thread may input the first request into the second model to obtain a plurality of third expert modules corresponding to the first request. The third expert module may refer to an expert module that needs to be preloaded into the first storage unit, so that the first thread can obtain the first expert module from the first storage unit when executing the second stage process. The second model can refer to the description of the corresponding part above, which will not be repeated here.

[0378] Regarding how the second thread utilizes the second model to obtain multiple third expert modules corresponding to the first request according to the first request, it can be implemented in the following manner.

[0379] As an implementation, the second thread may input the first request into the second model to obtain a plurality of third expert modules corresponding to the first request. For example, the first thread may input the first request into the second model to obtain a second score corresponding to each of the plurality of expert modules. According to the second scores corresponding to each of the plurality of expert modules, a plurality of third expert modules corresponding to the first request are obtained. The second score may be used to indicate a predicted activation score of the expert module. The predicted activation score may be used to indicate a probability of being activated.

[0380] How the second thread obtains multiple third expert modules corresponding to the first request according to the second scores corresponding to the multiple expert modules can be implemented in the following manner.

[0381] As an implementation method, the second thread can determine U second scores from the second scores corresponding to each of the multiple expert modules. The expert modules corresponding to each of the U second scores are determined as the third expert module. When the larger the second score is, the more likely the expert module is to be accessed, the U second scores can be the maximum U second scores. When the smaller the second score is, the greater the probability of the expert module being accessed, the U second scores can be the minimum U second scores. U can be an integer greater than 1 and less than R*S. U can be determined based on the storage capacity of the first storage unit, which is not limited here. For the description of R and S, please refer to the corresponding part above, which will not be repeated here.

[0382] In S1305 , the second thread retains the second expert modules in the second set corresponding to each of the plurality of third expert modules to obtain a first set.

[0383] According to an embodiment of the present application, the first set may include second expert modules in the second set corresponding to each of the plurality of third expert modules.

[0384] For any third expert module among the multiple third expert modules corresponding to the first request, in the process of the first thread loading at least one second expert module corresponding to at least one third word from the second storage unit to the first storage unit, if the second thread determines that there is a second expert module corresponding to the third expert module in the second set stored in the first storage unit, that is, if it is determined that the third expert module exists in the second set stored in the first storage unit, the second expert module corresponding to the third expert module in the first storage unit can be retained.

[0385] How the second thread reserves the second expert module corresponding to the third expert module in the second set can be implemented in the following manner.

[0386] As an implementation manner, the second thread may store the second expert module in the second set corresponding to the third expert module in a queue.

[0387] In addition, if the second thread determines that there is no second expert module corresponding to the third expert module in the second set stored in the first storage unit, the expert module corresponding to the third expert module in the second storage unit may be loaded from the second storage unit to the first storage unit. For example, the second thread may load the expert module corresponding to the third expert module in the second storage unit from the second storage unit to the queue of the first storage unit.

[0388] Therefore, when the first thread executes the second stage process, the second expert module stored in the first storage unit corresponds to the third expert module, that is, the first storage unit stores the third expert module.

[0389] How the first thread uses the first model to execute the second stage process can be implemented through the following S1306-S1316.

[0390] In S1306 , the first thread uses the gating module included in the first model to determine, according to the first word element corresponding to the first request, at least one first expert module corresponding to the first word element from the plurality of expert modules included in the first model.

[0391] According to an embodiment of the present application, the first word-element may be used to indicate an output word-element. For example, the first word-element may be the p-th output word-element described above. p may be an integer greater than or equal to 2 and less than or equal to P.

[0392] As an implementation, the first thread may use the gating module included in the first model to determine the third weights corresponding to each of the multiple expert modules included in the first model according to the first word element corresponding to the first request. For example, the first thread may use the gating module included in the first model to determine the third weights corresponding to each of the multiple expert modules included in the first model according to the first word element corresponding to the first request and at least one third word element corresponding to the first request. According to the third weights corresponding to each of the multiple expert modules included in the first model, at least one first expert module corresponding to the first word element is determined from the multiple expert modules. The third weight may be used to indicate the activation probability of the expert module.

[0393] According to an embodiment of the present application, as described above, the first model may include at least one first layer. The first layer may include a gating module and a plurality of expert modules. The at least one first expert module corresponding to the first word may include at least one first expert module corresponding to each of the at least one first layer.

[0394] Regarding how the first thread utilizes the gating module included in the first model and determines the third weights corresponding to each of the plurality of expert modules included in the first model according to the first word corresponding to the first request, this can be achieved in the following manner.

[0395] As an implementation, for any first layer in at least one first layer, the first layer may be used as the current layer, and the first thread may use the gating module corresponding to the current layer to determine the third weights corresponding to the multiple expert modules corresponding to the current layer according to the first word element corresponding to the first request. For example, the third weights corresponding to the multiple expert modules corresponding to the current layer may be determined according to the first word element corresponding to the first request and the second word element corresponding to the first request.

[0396] How the first thread determines at least one first expert module corresponding to the first word-gram from the multiple expert modules according to the third weights corresponding to the multiple expert modules included in the first model can be implemented in the following manner.

[0397] As an implementation manner, the first thread may determine at least one first expert module corresponding to the first word and the current layer from the multiple expert modules corresponding to the current layer according to the third weights respectively corresponding to the multiple expert modules corresponding to the current layer.

[0398] How the first thread determines at least one first expert module corresponding to the first word-gram and the current layer from a plurality of expert modules corresponding to the current layer may be implemented in the following manner.

[0399] The first method is based on the Top-T method. The first thread can determine T third weights from the third weights corresponding to each of the multiple expert modules corresponding to the current layer. The expert module corresponding to each of the T third weights is determined as the first expert module corresponding to the third word and the current layer. When the larger the third weight is, the greater the probability that the expert module is activated, the T third weights can be the maximum T third weights. When the smaller the third weight is, the greater the probability that the expert module is activated, the T third weights can be the minimum T third weights. T can be an integer greater than or equal to 1 and less than or equal to S. T can be configured according to actual business needs and is not limited here. For example, S=16. T=2.

[0400] The second method is based on the threshold method. For example, when the third weight is larger, the probability of the expert module being activated is larger, and the first thread can determine the third weight greater than or equal to the sixth threshold from the third weights corresponding to the multiple expert modules corresponding to the current layer. The expert module corresponding to the third weight greater than or equal to the seventh threshold is determined as the first expert module corresponding to the third word and the current layer. When the third weight is smaller, the probability of the expert module being activated is larger, and the first thread can determine the third weight less than or equal to the eighth threshold from the third weights corresponding to the multiple expert modules corresponding to the current layer. The expert module corresponding to the third weight less than or equal to the eighth threshold is determined as the first expert module corresponding to the third word and the current layer. The seventh threshold may be greater than the eighth threshold. The seventh threshold and the eighth threshold can be configured according to actual business needs and are not limited here.

[0401] In S1307 , the first thread determines whether there is a second expert module corresponding to the first expert module in the first set stored in the first storage unit. If so, S1308 - S1309 are executed; if not, S1310 is executed.

[0402] According to an embodiment of the present application, the second expert module may have a third identifier. The first expert module may have a first identifier. The first thread may determine whether there is a third identifier corresponding to the first identifier among the multiple third identifiers corresponding to the first set. In the case where it is determined that there is a third identifier corresponding to the first identifier, it can be determined that there is a second expert module corresponding to the first expert module among the multiple second expert modules. In the case where it is determined that there is no third identifier corresponding to the first identifier, it can be determined that there is no second expert module corresponding to the first expert module among the multiple second expert modules.

[0403] According to the embodiment of the present application, the second expert module corresponding to the first expert module can be understood as the second expert module is the first expert module. The third identifier corresponding to the first identifier can be understood as the third identifier is the first identifier.

[0404] At S1308 , the first thread obtains a second expert module corresponding to the first expert module from the first storage unit.

[0405] According to an embodiment of the present application, the first expert module may have a first identifier and the second expert module may have a third identifier.

[0406] As an implementation, the first thread may determine a third identifier corresponding to the first identifier from third identifiers corresponding to each of the plurality of second expert modules. The third expert module corresponding to the third identifier is determined as the second expert module corresponding to the first expert module. The first thread may obtain a model parameter corresponding to the third identifier from the first storage unit, thereby obtaining the second expert module corresponding to the first expert module.

[0407] In S1309 , the first thread processes the first word-gram using a second expert module corresponding to the first expert module.

[0408] According to an embodiment of the present application, the first thread can obtain the second expert module corresponding to the first expert module from the first storage unit when it is determined that there is a second expert module corresponding to the first expert module among the multiple second expert modules stored in the first storage unit. Thus, the first thread can use the second expert module corresponding to the first expert module to process the first word, and then the first thread implements the second stage process using the first model to obtain the inference result.

[0409] In S1310 , the first thread determines whether there is a fourth expert module corresponding to the first expert module among the multiple expert modules included in the first model, and if so, performs operation S1311 ; if not, performs S1314 - S1316 .

[0410] In S1311, the first thread determines whether there is a second expert module corresponding to the fourth expert module in the first set stored in the first storage unit, and if so, executes S1312-S1313; if not, executes S1314-S1316.

[0411] At S1312, the first thread obtains a second expert module corresponding to the fourth expert module from the first storage unit.

[0412] According to an embodiment of the present application, the fourth expert module may have a fifth identifier. The second expert module may have a third identifier.

[0413] As an implementation, the first thread may determine the third identifier corresponding to the fifth identifier from the third identifiers corresponding to the plurality of second expert modules corresponding to the first set. The second expert module corresponding to the third identifier is determined as the second expert module corresponding to the fourth expert module. The first thread may obtain the model parameters corresponding to the third identifier from the first storage unit, thereby obtaining the second expert module corresponding to the fourth expert module.

[0414] In S1313 , the first thread processes the first word-gram using the second expert module corresponding to the fourth expert module.

[0415] At S1314 , the first thread loads the expert module corresponding to the first expert module from the second storage unit to the first storage unit.

[0416] According to an embodiment of the present application, the expert module may have a second identifier. The first expert module may have a first identifier.

[0417] As an implementation method, the first thread can determine the second identifier corresponding to the first identifier from the second identifiers corresponding to the plurality of expert modules stored in the second storage unit. The expert module corresponding to the second identifier is determined as the expert module corresponding to the first expert module. The first thread can obtain the model parameters corresponding to the second identifier from the second storage unit, and load the model parameters corresponding to the second identifier from the second storage unit to the first storage unit, thereby the first thread loads the expert module corresponding to the first expert module from the second storage unit to the first storage unit.

[0418] At S1315 , the first thread obtains an expert module corresponding to the first expert module from the first storage unit.

[0419] At S1316 , the first thread processes the first word-gram using an expert module corresponding to the first expert module.

[0420] According to an embodiment of the present application, the fourth expert module may be used to replace the first expert module to process the first word. Fig.9A and Fig. 9B Chinese expert module 922.

[0421] The following describes how to determine whether there is a fourth expert module corresponding to the first expert module among the multiple expert modules included in the first model.

[0422] The first expert module may have a corresponding first weight. The fourth expert module may have a corresponding second weight. The first weight may be used to indicate the activation probability of the first expert module. The second weight may be used to indicate the activation probability of the fourth expert module. The second weight needs to meet the following conditions: the absolute value of the difference between the second weight and the first weight is less than or equal to the first threshold. Thus, the first weight may be greater than the second weight, or less than the second weight, as long as the absolute value of the difference between the first weight and the second weight is less than or equal to the first threshold. The first threshold may be configured according to actual business needs and is not limited here. For example, the first threshold may be less than or equal to 0.3.

[0423] The above has explained how the first thread determines at least one first expert module from multiple expert modules included in the first model according to the first request. On this basis, it is explained how to determine a fourth expert module corresponding to the first expert module.

[0424] The first method is based on the Top-T method. For example, the first thread can determine T third weights from the third weights corresponding to the multiple expert modules corresponding to the current layer. The T third weights can be determined as T first weights. For any first weight among the T first weights, the first thread can determine the second weight from other third weights according to the first weight and the first threshold. For example, the first thread can determine the first range according to the first weight and the first threshold. The first thread can determine the third weight belonging to the first range from other third weights. The third weight belonging to the first range is determined as the second weight. Optionally, the first thread can determine the third weight whose absolute value of the difference with the first weight is less than or equal to the first threshold from other third weights. The third weight whose absolute value of the difference with the first weight is less than or equal to the first threshold is determined as the second weight. Other third weights can refer to the third weights other than the first weight among the multiple third weights. Thus, the expert module corresponding to the second weight can be determined as the fourth expert module corresponding to the first expert module. In the case where the greater the third weight, the greater the probability of the expert module being activated, the T third weights can be the maximum T third weights. In the case where the smaller the third weight is, the greater the probability of the expert module being activated, the T third weights may be the minimum T third weights. T may be an integer greater than or equal to 1 and less than or equal to S. T may be configured according to actual business needs and is not limited here. For example, S=16. T=2.

[0425] It can be understood that if there is no third weight belonging to the first range or a third weight whose absolute value of difference with the first weight is less than or equal to the first threshold value among the other third weights, the first expert module does not have a corresponding fourth expert module. In addition, if there are multiple third weights belonging to the first range among the other third weights or there are multiple third weights whose absolute value of difference with the first weight is less than or equal to the first threshold value among the other third weights, the first expert module has multiple corresponding fourth expert modules.

[0426] The second method is based on the threshold method. For example, when the third weight is larger, the probability of the expert module being activated is larger. The first thread can determine a third weight greater than or equal to the seventh threshold from the third weights corresponding to the multiple expert modules corresponding to the current layer. The third weight greater than or equal to the seventh threshold can be determined as the first weight. When the third weight is smaller, the probability of the expert module being activated is larger. The first thread can determine a third weight less than or equal to the eighth threshold from the third weights corresponding to the multiple expert modules corresponding to the current layer. The third weight less than or equal to the eighth threshold can be determined as the first weight. The second weight can be determined from other third weights according to the first weight and the first threshold. For example, the first thread can determine the first range according to the first weight and the first threshold. The first thread can determine the third weight belonging to the first range from other third weights. The third weight belonging to the first range is determined as the second weight. Optionally, the first thread can determine a third weight whose absolute value of the difference with the first weight is less than or equal to the first threshold from other third weights. The third weight whose absolute value of the difference with the first weight is less than or equal to the first threshold is used as the second weight. Other third weights can refer to third weights other than the first weight among multiple third weights. Thus, the second expert module corresponding to the second weight can be determined as the fourth expert module corresponding to the first expert module. The seventh threshold can be greater than the eighth threshold. The seventh threshold and the eighth threshold can be configured according to actual business needs and are not limited here.

[0427] The following describes how the fourth expert module is stored in the first storage unit.

[0428] According to the above, the fourth expert module and the first expert module may correspond to the same request. For example, the fourth expert module and the first expert module correspond to the first request. In addition, the fourth expert module and the first expert module may correspond to different requests. For example, the fourth expert module corresponds to the third request, and the first expert module corresponds to the first request. The following describes the implementation of storing the fourth expert module in the first storage unit for the above two situations respectively.

[0429] As an implementation, if the fourth expert module and the first expert module correspond to the first request, the fourth expert module may be the third expert module determined by the second thread according to the first request using the second model when the first thread uses the first model to execute the first stage process corresponding to the first request, that is, the fourth expert module is the third expert module. Fig.9A The expert module 922. For how the second thread uses the second model to determine the third expert module according to the first request, please refer to the corresponding part of the above description, which will not be repeated here. In the above case, the fourth expert module can be stored in the first storage unit in the following manner.

[0430] For example, when the first thread uses the first model to execute the first stage process corresponding to the first request, the first thread can determine at least one second expert module corresponding to each of the at least one third word-element from the multiple expert modules included in the first model based on the at least one third word-element corresponding to the first request, and load the at least one second expert module corresponding to each of the at least one third word-element from the second storage unit to the first storage unit to obtain the second set. When the first thread executes the first stage process corresponding to the first request, the second thread can retain the second expert module corresponding to the third expert module in the first storage unit if it is determined that the second expert module corresponding to the third expert module exists in the second set stored in the first storage unit, that is, if it is determined that the third expert module exists in the second set stored in the first storage unit.

[0431] As another implementation, if the fourth expert module corresponds to the third request and the first expert module corresponds to the first request, then the fourth expert module may be stored in the first storage unit in the following manner in view of the different processing stages of the first request and the third request. Fig. 9B Chinese expert module 922.

[0432] As an implementation method, the fourth expert module may be the third expert module determined by the second thread when executing the preload process corresponding to the third request. For example, the fourth expert module may be the third expert module determined by the second thread according to the third request using the second model when the first thread executes the first stage process corresponding to the third request using the first model. As for how the second thread determines the third expert module according to the third request using the second model, a similar method to that of processing the first request may be adopted, which will not be described in detail here. In the above case, the fourth expert module may be stored in the first storage unit in the following manner.

[0433] For example, when the first thread uses the first model to execute the first stage process corresponding to the third request, the first thread can determine at least one second expert module corresponding to each of the at least one third word-element from the multiple expert modules included in the first model according to the at least one third word-element corresponding to the third request, and load the at least one second expert module corresponding to each of the at least one third word-element from the second storage unit to the first storage unit. The second set can include at least one second expert module corresponding to the third request. When the first thread executes the first stage process corresponding to the third request, if it is determined that there is a second expert module corresponding to the third expert module in the second set stored in the first storage unit, that is, if it is determined that there is a third expert module in the second set stored in the first storage unit, then the second expert module corresponding to the third expert module in the first storage unit can be retained. Thus, the first set can include at least one second expert module corresponding to the third request.

[0434] As another implementation, the fourth expert module may be a second expert module determined by the first thread using the first model to execute the first stage process corresponding to the third request. If the second expert module does not exist in the first storage unit, the first thread may load the expert module corresponding to the second expert module from the second storage unit to the first storage unit.

[0435] As another implementation, the fourth expert module may be the first expert module determined by the first thread using the first model to execute the second stage process corresponding to the third request. If the first storage unit does not contain a second expert module corresponding to the first expert module, the first thread may load the expert module corresponding to the first expert module from the second storage unit to the first storage unit.

[0436] The following describes how the first thread processes the first word.

[0437] The fourth expert module may have a fifth identification. The second expert module may have a third identification.

[0438] When the first thread determines that there is a fourth expert module corresponding to the first expert module among the multiple expert modules, the first thread can determine whether there is a third identifier corresponding to the fifth identifier among the multiple third identifiers. When it is determined that there is a third identifier corresponding to the fifth identifier, the first thread can obtain the second expert module corresponding to the fourth expert module from the first storage unit, thereby processing the first word unit using the second expert module corresponding to the fourth expert module. When it is determined that there is no third identifier corresponding to the fifth identifier, the first thread can load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtain the expert module corresponding to the first expert module from the first storage unit, and process the first word unit using the expert module corresponding to the first expert module.

[0439] When it is determined that there is no fourth expert module corresponding to the first expert module among multiple expert modules, the first thread may load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtain the expert module corresponding to the first expert module from the first storage unit, and use the expert module corresponding to the first expert module to process the first word.

[0440] Based on the above, the first thread may repeatedly execute S1306 - S1316 until an inference result is obtained.

[0441] Since the storage capacity (or memory capacity) of the first storage unit is limited, in the process of executing the data processing method based on the first thread and the second thread, there may be a situation where the first storage unit meets the first condition, thereby affecting the cache hit rate and resource utilization. The first condition can be used to trigger cache elimination. As an implementation method, the first condition can be used to indicate that the first storage unit is full.

[0442] To improve cache hit rate and resource utilization, the fourth thread can eliminate the second expert module from the first set stored in the first storage unit based on the cache elimination strategy when the first storage unit meets the first condition. The fourth thread can be executed in parallel with the first thread and the second thread.

[0443] The following describes how the fourth thread determines whether the first storage unit satisfies the first condition.

[0444] The first condition may include at least one of the following: the first capacity reaches the second threshold, the first number reaches the third threshold, the first usage rate reaches the fourth threshold, the first hit rate is less than or equal to the fifth threshold, or the second hit rate is greater than or equal to the ninth threshold.

[0445] The first capacity may be used to indicate the total storage size of the first set stored in the first storage unit. The second threshold may be determined according to the storage capacity of the first storage unit. For example, the second threshold may be the storage capacity of the first storage unit. Optionally, the second threshold may be the product of the first coefficient and the storage capacity of the first storage unit. The first coefficient may be greater than 0 and less than 1. The first coefficient may be configured according to actual business needs and is not limited here.

[0446] The first number may be used to indicate the total number of second expert modules in the first set. The third threshold may be determined based on the storage capacity of the first storage unit and the storage size of the second expert module. For example, the third threshold may be a ratio between the storage capacity of the first storage unit and the storage size of the second expert module.

[0447] The first usage rate may be used to indicate the ratio between the first capacity and the storage capacity. The fourth threshold may be configured according to actual business needs and is not limited here. For example, the fourth threshold may be 0.9.

[0448] The first hit rate may be used to indicate a ratio between the second number and the third number. The second number may be used to indicate a cache hit number corresponding to the first storage unit. The third number may be used to indicate a total access number corresponding to the first storage unit. The cache hit number may be used to indicate a number of times a first situation occurs. The first situation may be used to indicate that a first expert module exists in the first storage unit.

[0449] The second hit rate may be used to indicate a ratio between the fourth number and the third number. The fourth number may be used to indicate a number of cache misses corresponding to the first storage unit. The number of cache misses may be used to indicate a number of times the second situation occurs. The second situation may be used to indicate that the first expert module does not exist in the first storage unit.

[0450] The fifth threshold and the ninth threshold can be configured according to actual business needs and are not limited here. For example, the fifth threshold can be 0.9 and the ninth threshold can be 0.3.

[0451] The cache elimination strategy may include at least one of the following: first-in-first-out strategy, least recently used strategy, LRU-K strategy, least frequently used strategy, most recently used strategy, random elimination strategy, survival time strategy, second-level cache strategy, adaptive replacement cache strategy or low interactive reference interval strategy, etc.

[0452] The following describes how the fourth thread eliminates the second expert module from the first set stored in the first storage unit based on the cache elimination strategy. The eliminated second expert module may be referred to as a sixth expert module.

[0453] As an implementation manner, the plurality of second expert modules in the first set have respective corresponding first moments. The first moment can be used to indicate the moment when the second expert module is stored in the first storage unit.

[0454] The fourth thread can eliminate the sixth expert module from the first set stored in the first storage unit according to the first moment corresponding to each of the plurality of second expert modules when the first storage unit satisfies the first condition. For example, the fourth thread can determine the second moment from the first moment corresponding to each of the plurality of second expert modules. The second expert module corresponding to the second moment is determined as the sixth expert module, and the sixth expert module is eliminated. The second moment can be used to indicate the earliest first moment among the first moments corresponding to each of the plurality of second expert modules. That is, the fourth thread can eliminate the sixth expert module from the plurality of second expert modules when the first storage unit satisfies the first condition. The second moment is earlier than the first moment. The second moment can be used to indicate the storage moment when the sixth expert module is stored in the first storage unit. The first moment can be used to indicate the moment corresponding to any second expert module other than the sixth expert module among the plurality of second expert modules.

[0455] As an implementation, the first set stored in the first storage unit may be stored in a queue, that is, the first storage unit stores a queue, and the queue may include multiple expert modules in the first set. The fourth thread may eliminate the second expert module at the head of the queue when the first storage unit meets the first condition. The second expert module at the head of the queue may refer to the sixth expert module. The queue may be implemented using an array or a linked list.

[0456] As another implementation, the plurality of second expert modules in the first set have respective corresponding third times. The third time may be used to indicate the time when the second expert module is most recently accessed.

[0457] The fourth thread can eliminate the sixth expert module from the first set stored in the first storage unit according to the third moments corresponding to each of the plurality of second expert modules when the first storage unit satisfies the first condition. For example, the fourth thread can determine the fourth moment from the third moments corresponding to each of the plurality of second expert modules. The second expert module corresponding to the fourth moment is determined as the sixth expert module. The fourth moment can be used to indicate the earliest third moment among the third moments corresponding to each of the plurality of second expert modules or the fourth moment can be used to indicate the latest third moment among the third moments corresponding to each of the plurality of second expert modules. That is, the fourth thread can eliminate the sixth expert module from the plurality of second expert modules when the first storage unit satisfies the first condition. The fourth moment can be used to indicate the moment when the sixth expert module is most recently accessed. The fourth moment is earlier than the third moment or the fourth moment is later than the third moment. The third moment can be used to indicate the moment corresponding to any second expert module other than the sixth expert module among the plurality of second expert modules.

[0458] As another implementation, the plurality of second expert modules in the first set have respective corresponding first frequencies. The first frequency may be used to indicate the frequency at which the second expert modules are accessed.

[0459] The fourth thread may eliminate the sixth expert module from the first set stored in the first storage unit according to the first frequencies corresponding to the plurality of second expert modules respectively, if the first storage unit satisfies the first condition. For example, the fourth thread may determine the second frequency from the first frequencies corresponding to the plurality of second expert modules respectively. The second expert module corresponding to the second frequency is determined as the sixth expert module, and the sixth expert module is eliminated. The second frequency may be used to indicate the smallest first frequency among the first frequencies corresponding to the plurality of second expert modules respectively. That is, the fourth thread may eliminate the sixth expert module from the plurality of second expert modules if the first storage unit satisfies the first condition. The second frequency is less than the first frequency. The second frequency may be used to indicate the frequency at which the sixth expert module is accessed. The first frequency may be used to indicate the frequency corresponding to any second expert module other than the sixth expert module among the plurality of second expert modules.

[0460] When the first thread executes the first stage process, the fourth thread can, based on the cache elimination strategy, eliminate the second expert module from the first set stored in the first storage unit if the first storage unit satisfies the first condition, so that the first thread can load the expert module corresponding to the second expert module from the second storage unit to the first storage unit.

[0461] When the first thread executes the second stage process, if the first storage unit satisfies the first condition, the fourth thread can eliminate the second expert module from the first set stored in the first storage unit based on the cache elimination strategy, so that the first thread can load the expert module corresponding to the first expert module from the second storage unit to the first storage unit or the third thread can load the expert module corresponding to the first expert module from the second storage unit to the first storage unit.

[0462] For ease of understanding, the following takes the first model deployed on the intelligent assistant of the terminal device as an example. Figure 14A-14E right Fig.13 The embodiments described are described.

[0463] Fig.14A A schematic diagram of an application scenario of the data processing method provided in an embodiment of the present application.

[0464] like Fig.14A As shown, the display interface of the end-side device displays a dialog box for a conversation with the intelligent assistant. The first request "In which city is FD University located" is entered in the dialog box. The intelligent assistant can use Fig.13 The data processing method processes the first request and obtains the inference result "FD University is located in SH".

[0465] In addition, there are multiple ways to start the smart assistant. For example, the smart assistant can be started by long pressing the navigation bar at the bottom of the terminal device. The embodiment of the present application does not limit the way to start the smart assistant.

[0466] Combine the following Figure 14B-14E right Fig.14A How to use intelligent assistants Fig.13 The data processing method is described.

[0467] Fig. 14B A schematic diagram of storing the expert module of the first model in the second storage unit provided in an embodiment of the present application.

[0468] like Fig. 14B As shown, the first model 1410 may include a first layer 1411 and a first layer 1412. The first layer 1411 may include a gating module 1411_1 and 16 expert modules. The 16 expert modules may include expert modules 1411_2_1, ..., expert modules 1411_2_s, ..., and expert modules 1411_2_16. The first layer 1412 may include a gating module 1412_1 and 16 expert modules. The 16 expert modules may include expert modules 1412_2_1, ..., expert modules 1412_2_s, ..., and expert modules 1412_2_16. s may be an integer greater than or equal to 1 and less than or equal to 16. As an implementation, Fig. 14B The first model 1410 may be Figure 6 The first model 600 is obtained when R=2 and S=16. In addition, the first model 1410 and the first model 600 may be different.

[0469] The 32 expert modules of the first model can be stored in the second storage unit. In addition, the second storage unit can also store a non-mixed expert layer.

[0470] exist Fig. 14B On the basis of Fig. 14C A schematic diagram of the principle of another data processing method provided in an embodiment of the present application.

[0471] like Fig. 14C As shown, the first thread can tokenize the first request "In which city is FD University located" to obtain multiple third tokens corresponding to the first request, namely, the third token ["FD"], the third token ["university"], the third token ["located in"], the third token ["which"] and the third token ["city"]. The first thread can use the first model 1410 to execute the first stage process (i.e., the pre-filling stage process) to obtain the second token (i.e., the first output token), that is, the first thread can use the first model to obtain the second token according to the multiple third tokens corresponding to the first request. The first thread can use the first model 1410 to execute the second stage process (i.e., the decoding stage process), that is, the first thread can use the first model to obtain the inference result according to the first request and the second token. The inference result can be determined based on multiple first tokens. The third token can be used to indicate the input token. The first token can be used to indicate the output token. The second token can also be called the first token.

[0472] When the first thread executes the first phase process using the first model 1410 , the second thread may execute the preloading process using the second model 1420 .

[0473] exist Fig. 14C Based on the following Fig.14D , it is described how the first thread uses the first model 1410 to execute the first stage process and how the second thread uses the second model 1420 to execute the preloading process.

[0474] Fig.14D A schematic diagram of an embodiment of the present application showing that a first thread uses a first model to execute a first phase process and a second thread uses a second model to execute a preloading process.

[0475] like Fig.14DAs shown, the first thread can implement the execution of the first-stage process using the first model in the following manner. For example, using the first model 1410, according to the first request, a plurality of second expert modules corresponding to each of the plurality of third tokens are obtained. For any third token among the plurality of third tokens, the first thread can load the plurality of second expert modules corresponding to the third token from the second storage unit to the first storage unit. Thus, a second set is obtained.

[0476] When the first thread is executing the first-stage process using the first model 1410, the second thread can use the second model 1420 to obtain a plurality of third expert modules according to the first request. For any one of the plurality of third expert modules corresponding to the first request, during the process that the first thread loads the plurality of second expert modules corresponding to each of the plurality of third tokens from the second storage unit to the first storage unit, if the second thread determines that there is a second expert module corresponding to the third expert module in the second set stored in the first storage unit, that is, if it determines that the third expert module exists in the second set stored in the first storage unit, it can retain the second expert module corresponding to the third expert module in the first storage unit. Thus, the first storage unit stores a first set. The first set may include the second expert modules corresponding to the third expert modules. The second expert modules in the first set can be used to obtain the second expert module corresponding to the first expert module from the plurality of second expert modules when the first thread uses the first model to execute the second-stage process.

[0477] Based on Figure 14B-Figure 14D , taking the first token ["located at"] as an example, in combination with Fig.14E it is described how the first thread uses the first model to execute the second-stage process.

[0478] Fig.14E FIG. is a schematic diagram of the first thread using the first model to execute the second-stage process provided by the embodiments of the present application.

[0479] As Fig.14EAs shown, the first thread uses the gating module 1411_1 to determine multiple first expert modules corresponding to the first word [“located at”] and the first layer 1411 from the 16 expert modules included in the first layer 1411 according to the first word [“located at”], the first word [“FD”], the first word [“university”] and the multiple third words corresponding to the first request. The multiple first expert modules corresponding to the first word [“located at”] and the first layer 1411 may include expert module 1411_2_1 and expert module 1411_2_16. The first thread determines that there is a second expert module corresponding to the first expert module (i.e., expert module 1411_2_1) and a second expert module corresponding to the first expert module (i.e., expert module 1411_2_16) in the first set stored in the first storage unit, so the first thread can obtain the above-mentioned second expert module from the first storage unit and use the above-mentioned second expert module to process the first word.

[0480] The first thread can use the gating module 1412_1 to determine multiple first expert modules corresponding to the first word [“located at”] and the first layer 1412 from the 16 expert modules included in the first layer 1412 according to the second feature. The multiple first expert modules corresponding to the first word [“located at”] and the first layer 1412 may include expert module 1412_2_1 and expert module 1412_2_16. The second feature can be obtained by the first thread using the first layer 1411 according to the first word [“located at”], the first word [“FD”], the first word [“university”] and multiple third words corresponding to the first request. The first thread can determine that there is a second expert module corresponding to the first expert module (i.e., expert module 1412_2_1) and a second expert module corresponding to the first expert module (i.e., expert module 1412_2_16) in the first set stored in the first storage unit, so the first thread can obtain the above-mentioned second expert module from the first storage unit and use the above-mentioned second expert module to process the first word.

[0481] Fig.15 A flowchart of another data processing method provided in an embodiment of the present application. The method can be applied to the first thread, the second thread, and the third thread.

[0482] like Fig.15 As shown, the method includes S1501-S1515.

[0483] In S1501, the first thread determines, according to at least one third word-gram corresponding to the first request, at least one second expert module corresponding to each of the at least one third word-grams from a plurality of expert modules included in the first model.

[0484] In S1502, the first thread loads at least one second expert module corresponding to each of the at least one third word-grams from the second storage unit to the first storage unit to obtain a second set.

[0485] In S1503, the first thread uses the second set and obtains a second word-gram corresponding to the first request according to at least one third word-gram.

[0486] In S1504, the second thread uses the second model to obtain multiple third expert modules corresponding to the first request according to the first request.

[0487] In S1505 , the second thread reserves the second expert module corresponding to the third expert module in the second set to obtain a first set.

[0488] At S1506 , the first thread uses the gating module included in the first model to determine, according to the first word element corresponding to the first request, at least one first expert module corresponding to the first word element from the plurality of expert modules included in the first model.

[0489] In S1507 , the first thread determines whether there is a second expert module corresponding to the first expert module in the first set stored in the first storage unit. If so, S1508 - S1509 are executed; if not, S1510 is executed.

[0490] At S1508 , the first thread obtains a second expert module corresponding to the first expert module from the first storage unit.

[0491] In S1509 , the first thread processes the first word-gram using a second expert module corresponding to the first expert module.

[0492] In S1510, the first thread determines whether there is a fourth expert module corresponding to the first expert module among the multiple expert modules included in the first model. If so, S1511 is executed; if not, S1514-S1515 are executed.

[0493] In S1511, the first thread determines whether there is a second expert module corresponding to the fourth expert module in the first set, and if so, executes S1512-S1513; if not, executes S1514-S1515.

[0494] At S1512, the first thread obtains a second expert module corresponding to the fourth expert module from the first storage unit.

[0495] In S1513, the first thread processes the first word using the second expert module corresponding to the fourth expert module.

[0496] Since the absolute value of the difference between the second weight corresponding to the fourth expert module and the first weight corresponding to the first expert module is less than or equal to the first threshold, the second weight can be used to indicate the activation probability of the fourth expert module, and the first weight can be used to indicate the activation probability of the first expert module. Therefore, it can be explained that the prediction accuracy of the fourth expert module is relatively close to the prediction accuracy of the first expert module, and thus, the fourth expert module can replace the first expert module.

[0497] Based on this, when the second expert module corresponding to the first expert module does not exist in the first storage unit and the second expert module corresponding to the fourth expert module exists in the first storage unit, the first thread can directly obtain the second expert module corresponding to the fourth expert module from the first storage unit to use the second expert module corresponding to the fourth expert module to process the first word, thereby further improving the cache hit rate. In addition, since there is no need to first load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, and then obtain the expert module corresponding to the first expert module, and use the expert module corresponding to the first expert module to process the first word, the I / O delay between the second storage unit and the first storage unit is reduced, thereby improving the reasoning efficiency of the first model.

[0498] At S1514, the third thread obtains an expert module corresponding to the first expert module from the second storage unit.

[0499] At S1515 , the third thread processes the first word-gram using an expert module corresponding to the first expert module.

[0500] According to the embodiment of the present application, the description of S1501-S1512 can refer to the above description of Fig.13 The description will not be repeated here.

[0501] and Fig.13 The difference between the illustrated embodiments is that, in the present embodiment, when the first thread determines that the second expert module corresponding to the fourth expert module does not exist in the first set stored in the first storage unit, the first thread uses the third thread to process the first word unit, that is, the third thread can obtain the expert module corresponding to the first expert module from the second storage unit, and use the expert module corresponding to the first expert module to process the first word unit. The third speed can be greater than the fourth speed. The third speed can be used to indicate the speed at which the third thread accesses the second storage unit. The fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit.

[0502] Since the first thread can be used to indicate the accelerator thread, the third thread can be used to indicate the processor thread, the first storage unit can be used to indicate the accelerator memory, the second storage unit can be used to indicate the processor memory, the third speed can be used to indicate the speed at which the third thread accesses the second storage unit, and the fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit, the third access speed is greater than the fourth access speed. When the first thread determines that the second expert module corresponding to the fourth expert module does not exist in the first set stored in the first storage unit, the third thread is used to obtain the expert module corresponding to the first expert module from the second storage unit, and the expert module corresponding to the first expert module is used to process the first word, without the first thread first loading the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtaining the expert module corresponding to the first expert module from the first storage unit, and then using the expert module corresponding to the first expert module to process the first word, thereby reducing the I / O delay between the second storage unit and the first storage unit and improving the reasoning efficiency of the first model.

[0503] Fig.16 A flowchart of another data processing method provided in an embodiment of the present application. The method can be applied to the first thread and the second thread.

[0504] like Fig.16 As shown, the method includes S1601-S1612.

[0505] In S1601, the first thread determines, according to at least one third word-gram corresponding to the first request, at least one second expert module corresponding to each of the at least one third word-grams from a plurality of expert modules included in the first model.

[0506] In S1602, the first thread loads at least one second expert module corresponding to each of the at least one third word-grams from the second storage unit to the first storage unit to obtain a second set.

[0507] In S1603, the first thread uses the second set and obtains a second word-gram corresponding to the first request according to at least one third word-gram.

[0508] In S1604, the second thread uses the second model to obtain multiple third expert modules corresponding to the first request according to the first request.

[0509] In S1605, the second thread reserves the second expert module corresponding to the third expert module in the second set to obtain a first set.

[0510] At S1606, the first thread uses the gating module included in the first model to determine, according to the first word element corresponding to the first request, at least one first expert module corresponding to the first word element from the plurality of expert modules included in the first model.

[0511] In S1607, the first thread determines whether there is a second expert module corresponding to the first expert module in the first set stored in the first storage unit, and if so, executes S1608-S1609; if not, executes S1610-S1612.

[0512] At S1608 , the first thread obtains a second expert module corresponding to the first expert module from the first storage unit.

[0513] In S1609 , the first thread processes the first word-gram using a second expert module corresponding to the first expert module.

[0514] At S1610 , the first thread loads an expert module corresponding to the first expert module from the second storage unit to the first storage unit.

[0515] At S1611 , the first thread obtains an expert module corresponding to the first expert module from the first storage unit.

[0516] At S1612 , the first thread processes the first word-gram using an expert module corresponding to the first expert module.

[0517] According to the embodiment of the present application, the description of S1601-S1609 can refer to the above description of Fig.13 The description will not be repeated here.

[0518] and Fig.13 The difference between the embodiments shown is that, in this embodiment, when the first thread determines that there is no second expert module corresponding to the first expert module in the first set stored in the first storage unit, S1610-S1612 is executed. Fig.13 S1314-S1316 shown.

[0519] Fig.17 A flowchart of another data processing method provided in an embodiment of the present application. The method can be applied to the first thread, the second thread, and the third thread.

[0520] like Fig.17 As shown, the method includes S1701-S1711.

[0521] In S1701, the first thread determines, according to at least one third word-gram corresponding to the first request, at least one second expert module corresponding to each of the at least one third word-grams from a plurality of expert modules included in the first model.

[0522] In S1702, the first thread loads at least one second expert module corresponding to each of the at least one third word-grams from the second storage unit to the first storage unit to obtain a second set.

[0523] In S1703, the first thread uses the second set and obtains a second word-gram corresponding to the first request according to at least one third word-gram.

[0524] In S1704, the second thread uses the second model to obtain multiple third expert modules corresponding to the first request according to the first request.

[0525] In S1705, the second thread reserves the second expert module corresponding to the third expert module in the second set to obtain a first set.

[0526] At S1706 , the first thread uses the gating module included in the first model to determine, according to the first word element corresponding to the first request, at least one first expert module corresponding to the first word element from the plurality of expert modules included in the first model.

[0527] In S1707, the first thread determines whether there is a second expert module corresponding to the first expert module in the first set stored in the first storage unit, and if so, executes S1708-S1709; if not, executes S1710-S1711.

[0528] At S1708 , the first thread obtains a second expert module corresponding to the first expert module from the first storage unit.

[0529] In S1709 , the first thread processes the first word-gram using a second expert module corresponding to the first expert module.

[0530] At S1710 , the third thread obtains an expert module corresponding to the first expert module from the second storage unit.

[0531] At S1711 , the third thread processes the first word-gram using an expert module corresponding to the first expert module.

[0532] According to the embodiment of the present application, the description of S1701-S1709 can refer to the above description of Fig.13 The description will not be repeated here.

[0533] and Fig.13 The difference between the illustrated embodiments is that, in this embodiment, when the first thread determines that there is no second expert module corresponding to the first expert module in the first set stored in the first storage unit, the first thread uses the third thread to obtain the expert module corresponding to the first expert module from the second storage unit, and uses the expert module corresponding to the first expert module to process the first word.

[0534] Since the first thread can be used to indicate the accelerator thread, the third thread can be used to indicate the processor thread, the first storage unit can be used to indicate the accelerator memory, the second storage unit can be used to indicate the processor memory, the third speed can be used to indicate the speed at which the third thread accesses the second storage unit, and the fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit, the third access speed is greater than the fourth access speed. When the first thread determines that there is no second expert module corresponding to the first expert module in the first set, the third thread is used to obtain the expert module corresponding to the first expert module from the second storage unit, and the expert module corresponding to the first expert module is used to process the first word, without the first thread first loading the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtaining the expert module corresponding to the first expert module from the first storage unit, and then using the expert module corresponding to the first expert module to process the first word, thereby reducing the I / O delay between the second storage unit and the first storage unit and improving the reasoning efficiency of the first model.

[0535] It should be noted that the first layer 600_r, the first layer 900 and the first layer 1000 may be different from each other, may be partially the same, or may be completely the same.

[0536] The first layer 1411, the first layer 900 and the first layer 1000 may be different from each other, or may be partially the same, or may be completely the same. The first layer 1412, the first layer 900 and the first layer 1000 may be different from each other, or may be partially the same, or may be completely the same.

[0537] Expert module 420_q, expert module 612_r_s, expert module 921 and expert module 1021 may be different from each other, or may be partially the same, or may be completely the same. Expert module 420_q, expert module 612_r_s, expert module 922 and expert module 1022 may be different from each other, or may be partially the same, or may be completely the same. Expert module 420_q, expert module 612_r_s, expert module 923 and expert module 1023 may be different from each other, or may be partially the same, or may be completely the same. Expert module 420_q, expert module 612_r_s, expert module 924 and expert module 1024 may be different from each other, or may be partially the same, or may be completely the same. Gating module 611_r, gating module 910 and gating module 1010 may be different from each other, or may be partially the same, or may be completely the same. q∈{1,……,Q}. r∈{1,……,R}. s∈{1,……,S}.

[0538] Expert module 1411_2_s, expert module 420_q, expert module 921 and expert module 1021 may be different from each other, or may be partially the same, or may be completely the same. Expert module 1411_2_s, expert module 420_q, expert module 922 and expert module 1022 may be different from each other, or may be partially the same, or may be completely the same. Expert module 1411_2_s, expert module 420_q, expert module 923 and expert module 1023 may be different from each other, or may be partially the same, or may be completely the same. Expert module 1411_2_s, expert module 420_q, expert module 924 and expert module 1024 may be different from each other, or may be partially the same, or may be completely the same. s∈{1,……,16}.

[0539] Expert module 1412_2_s, expert module 420_q, expert module 921 and expert module 1021 may be different from each other, or may be partially the same, or may be completely the same. Expert module 1412_2_s, expert module 420_q, expert module 922 and expert module 1022 may be different from each other, or may be partially the same, or may be completely the same. Expert module 1412_2_s, expert module 420_q, expert module 923 and expert module 1023 may be different from each other, or may be partially the same, or may be completely the same. Expert module 1412_2_s, expert module 420_q, expert module 924 and expert module 1024 may be different from each other, or may be partially the same, or may be completely the same.

[0540] Fig.18 A flowchart of another data processing method provided in an embodiment of the present application.

[0541] like Fig.18 As shown, the method includes S1810-S1820.

[0542] At S1810, when the first thread executes the second stage process using the first model, if the first storage unit has a second expert module corresponding to the first expert module, the first thread obtains the second expert module corresponding to the first expert module from the first storage unit. The first expert module is determined by the first thread from a plurality of expert modules included in the first model according to the first word corresponding to the first request.

[0543] According to an embodiment of the present application, the first expert module may be determined by the first thread from the multiple expert modules included in the first model according to the third weights corresponding to the multiple expert modules included in the first model. The third weight may be obtained by the first thread calling the gating module included in the first model and using the gating module according to the first word corresponding to the first request.

[0544] It should be noted that for the description of how the first thread determines the first expert module, how the first thread determines whether the first expert module exists in the first storage unit, and how the first thread obtains the first expert module from the first storage unit, please refer to the corresponding parts above and will not be repeated here.

[0545] At S1820 , the first thread processes the first word-gram using a second expert module corresponding to the first expert module.

[0546] According to an embodiment of the present application, the first set stored in the first storage unit may be determined by the second thread according to the first request when the first thread executes the first stage process using the first model. The first set may include multiple second expert modules. The first stage process may be used to indicate that the first request is processed using the first model to obtain the second word element. The second word element may be used to indicate the first output word element corresponding to the first request. The second stage process may be used to indicate that the inference result is obtained according to the first request and the second word element using the first model.

[0547] It should be noted that, for the description of how the first storage unit stores multiple second expert modules, please refer to the corresponding part above, which will not be repeated here.

[0548] According to an embodiment of the present application, since the multiple second expert modules of the first storage unit are determined by the second thread according to the first request when the first thread executes the first stage process using the first model, the parallel execution of the preloading process using the second thread and the first stage process using the first thread is realized, thereby hiding the I / O delay as much as possible, thereby reducing the I / O delay and improving the reasoning efficiency of the first model. And since the first request has an expert preference, and the multiple second expert modules stored in the first storage unit can be determined according to the first request, therefore, when the first thread executes the second stage process using the first model, the first thread can directly obtain the second expert module corresponding to the first expert module from the first storage unit, thereby improving the cache hit rate of the first thread obtaining the first expert module from the first storage unit, thereby improving the reasoning efficiency of the first model. How the first storage unit stores the first set can be achieved in the following way.

[0549] As an implementation, when the first thread uses the first model to execute the first stage process, the second thread can determine multiple third expert modules from the multiple expert modules included in the first model according to the first request. The second thread can determine the second expert modules corresponding to each of the multiple third expert modules from the second set. The second thread can retain the second expert modules corresponding to each of the multiple third expert modules in the second set to obtain the first set. That is, the first set stored in the first storage unit can be obtained by the second thread retaining the second expert modules corresponding to each of the multiple third expert modules in the second set when the first thread uses the first model to execute the first stage process. The multiple third expert modules can be determined by the second thread from the multiple expert modules included in the first model according to the first request.

[0550] The second set may be stored in the first storage unit. The second set may include a plurality of second expert modules. The second set may be loaded from the second storage unit to the first storage unit by the first thread.

[0551] It should be noted that, for the description of the third expert module, the first set and the second set, please refer to the corresponding parts above and will not be repeated here.

[0552] Since the first set stored in the first storage unit can be obtained by retaining the second expert modules corresponding to the third expert modules in the second set when the first thread executes the first stage process using the first model, the parallel execution of the preloading process using the second thread and the first stage process using the first thread is realized, thereby hiding the I / O delay as much as possible, thereby reducing the I / O delay and improving the reasoning efficiency of the first model. And since the first request has an expert preference, and the third expert module can be determined by the second thread according to the first request when the first thread executes the first stage process using the first model, therefore, when the first thread executes the second stage process using the first model, the first thread can directly obtain the second expert module corresponding to the first expert module from the first storage unit, thereby improving the cache hit rate of the first thread obtaining the first expert module from the first storage unit, thereby improving the reasoning efficiency of the first model.

[0553] How the second thread determines multiple third expert modules from multiple expert modules according to the first request can be implemented in the following manner.

[0554] As an implementation, the second thread may use the second model to determine multiple third expert modules from multiple expert modules included in the first model according to the first request. The second model may be trained using the second request and multiple first scores corresponding to the second request. The first score may be used to indicate an activation score of the second expert module.

[0555] It should be noted that, for the description of the second model, the second request and the first score, please refer to the corresponding parts above and will not be repeated here.

[0556] How the first thread stores the second set can be implemented in the following manner.

[0557] As an implementation, the first thread may determine at least one third word element according to the first request. The first thread may determine at least one second expert module corresponding to each of the at least one third word elements from the multiple expert modules included in the first model. The first thread may load at least one second expert module corresponding to each of the at least one third word elements from the second storage unit to the first storage unit to obtain a second set. That is, the second set may be the result of the first thread loading at least one second expert module corresponding to each of the at least one third word elements from the second storage unit to the first storage unit. The at least one second expert module corresponding to each of the at least one third word elements may be determined by the first thread according to the at least one third word element from the multiple expert modules included in the first model. The at least one third word element may be determined by the first thread according to the first request.

[0558] It should be noted that, for the description of the second expert module, the third word unit and the second set, please refer to the corresponding parts above and will not be repeated here.

[0559] How the second thread determines the third expert module according to the first request can be implemented in the following manner.

[0560] As an implementation, the first set stored in the first storage unit may be determined by the second thread according to the first request using the second model when the first thread executes the first stage process using the first model. That is, the second thread may determine the first set stored in the first storage unit according to the first request using the second model when the first thread executes the first stage process using the first model. The second model may be trained using the second request and a plurality of first scores corresponding to the second request. The first score may be used to indicate the activation score of the expert module.

[0561] How the second model uses the second request and the multiple second scores corresponding to the second request to train the second model can be implemented in the following manner.

[0562] As an implementation, the second model may be obtained by adjusting the model parameters of the second model according to the loss function value. The loss function value may be obtained based on the loss function according to the first scores and the second scores corresponding to the plurality of expert modules. The second scores corresponding to the plurality of expert modules may be obtained by inputting the second request into the second model. The second score may be used to indicate the predicted activation score of the expert module.

[0563] Regarding how to obtain the loss function value based on the loss function according to the first scores and the second scores corresponding to each of the multiple expert modules, it can be achieved in the following manner.

[0564] As an implementation, the loss function value may be obtained according to the loss function values ​​corresponding to each of the multiple expert modules. The loss function value corresponding to the expert module may be obtained by inputting the first score and the second score corresponding to the expert module into the loss function.

[0565] Since the second model can be trained based on the expert preference of the request, that is, the second model can be trained using the second request and multiple first scores corresponding to the second request, the first score can be used to indicate the activation score of the expert module, thereby improving the expert prediction accuracy of the second model. On this basis, the first set stored in the first storage unit can be obtained by the second thread inputting the first request into the second model when the first thread executes the first stage process using the first model. Therefore, the first thread can directly obtain the second expert module corresponding to the first expert module from the first storage unit, thereby improving the cache hit rate of the first thread obtaining the first expert module from the first storage unit, thereby improving the reasoning efficiency of the first model.

[0566] How the first thread implements data processing when the second expert module corresponding to the first expert module does not exist in the first storage unit can be implemented in the following manner.

[0567] As an implementation, the first thread may obtain the second expert module corresponding to the fourth expert module from the first storage unit when the first storage unit does not contain the second expert module corresponding to the first expert module and the first storage unit contains the second expert module corresponding to the fourth expert module. The first thread may process the first word using the second expert module corresponding to the fourth expert module.

[0568] According to an embodiment of the present application, the first expert module may have a corresponding first weight. The first weight may be used to indicate the activation probability of the first expert module. The fourth expert module may have a corresponding second weight. The second weight may be used to indicate the activation probability of the fourth expert module. The absolute value of the difference between the first weight and the second weight may be less than or equal to the first threshold.

[0569] Since the absolute value of the difference between the second weight corresponding to the fourth expert module and the first weight corresponding to the first expert module is less than or equal to the first threshold, the second weight can be used to indicate the activation probability of the fourth expert module, and the first weight can be used to indicate the activation probability of the first expert module. Therefore, it can be shown that the prediction accuracy of the fourth expert module is close to the prediction accuracy of the first expert module, and thus, the fourth expert module can replace the first expert module. Based on this, when the second expert module corresponding to the first expert module does not exist in the first storage unit and the second expert module corresponding to the fourth expert module exists in the first storage unit, the first thread can directly obtain the second expert module corresponding to the fourth expert module from the first storage unit to use the second expert module corresponding to the fourth expert module to process the first word unit, thereby improving the cache hit rate. In addition, since there is no need to first load the expert module corresponding to the first expert module from the second storage unit to the first storage unit, and then obtain the expert module corresponding to the first expert module, and use the expert module corresponding to the first expert module to process the first word unit, the I / O delay between the second storage unit and the first storage unit is reduced, thereby improving the reasoning efficiency of the first model.

[0570] As an implementation manner, the fourth expert module may be the second expert module in the first set corresponding to the first request.

[0571] As another implementation, the first set may further include a plurality of second expert modules corresponding to the third request. The fourth expert module may be a second expert module in the first set corresponding to the third request. The third request may be different from the first request.

[0572] It should be noted that, for the description of how the fourth expert module is stored in the first storage unit, please refer to the corresponding part above, which will not be repeated here.

[0573] Since the fourth expert module can be the second expert module corresponding to the first request or the second expert module corresponding to the third request, the diversity of sources of the fourth expert module is improved, which helps to further reduce I / O delay.

[0574] How the first thread determines the first expert module and the fourth expert module can be implemented in the following manner.

[0575] As an implementation, the first thread may call the gating module included in the first model, and use the gating module to obtain the third weight corresponding to each of the multiple expert modules according to the first word corresponding to the first request. The first thread may determine the first weight according to the third weight corresponding to each of the multiple expert modules. The first thread may determine the second weight from the third weights corresponding to each of the multiple expert modules according to the first weight and the first threshold, that is, the first weight may be determined by the first thread according to the third weight corresponding to each of the multiple expert modules included in the first model. The third weight corresponding to each of the multiple expert modules included in the first model may be obtained by the first thread calling the gating module included in the first model and using the gating module according to the first word corresponding to the first request. The second weight may be determined by the first thread according to the first weight and the first threshold from the third weight corresponding to each of the multiple expert modules included in the first model.

[0576] It should be noted that for the description of how the first thread determines the first weight according to the third weights corresponding to each of the multiple expert modules included in the first model, and how the first thread determines the second weight from the third weights corresponding to each of the multiple expert modules included in the first model based on the first weight and the second weight, please refer to the corresponding parts above and will not be repeated here.

[0577] Since the second weight corresponding to the fourth expert module can be determined from the third weights corresponding to the plurality of expert modules included in the first model according to the first threshold and the first weight corresponding to the first expert module, the second weight can be used to indicate the activation probability of the fourth expert module, and the first weight can be used to indicate the activation probability of the first expert module, thus achieving that the determined activation probability of the fourth expert module is closer to the activation probability of the first expert module. Thus, the fourth expert module can replace the first expert module to process the first word.

[0578] As another implementation method, the first thread may load the first expert module from the second storage unit to the first storage unit, obtain the second expert module corresponding to the first expert module from the first storage unit, and process the first word using the second expert module corresponding to the first expert module when there is no second expert module corresponding to the first expert module in the first storage unit and there is no fourth expert module corresponding to the first expert module in the first storage unit.

[0579] As another implementation, when the first storage unit does not contain the first expert module and the first storage unit does not contain the second expert module corresponding to the fourth expert module, the third thread may obtain the expert module corresponding to the first expert module from the second storage unit, and use the expert module corresponding to the first expert module to process the first word. The third thread may be used to indicate the processor thread. The first thread may be used to indicate the accelerator thread.

[0580] Since the first thread can be used to indicate the accelerator thread, the third thread can be used to indicate the processor thread, the first storage unit can be used to indicate the accelerator memory, the second storage unit can be used to indicate the processor memory, the third speed can be used to indicate the speed at which the third thread accesses the second storage unit, and the fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit, the third access speed is greater than the fourth access speed. When the first thread determines that the first storage unit does not contain the second expert module corresponding to the first expert module and does not contain the second expert module corresponding to the fourth expert module, the third thread is used to obtain the expert module corresponding to the first expert module from the second storage unit, and the expert module corresponding to the first expert module is used to process the first word, without the first thread first loading the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtaining the expert module corresponding to the first expert module from the first storage unit, and then using the expert module corresponding to the first expert module to process the first word, thereby reducing the I / O delay between the second storage unit and the first storage unit and improving the reasoning efficiency of the first model.

[0581] As another implementation method, the first thread can load the expert module corresponding to the first expert module from the second storage unit to the first storage unit when there is no second expert module corresponding to the first expert module in the first storage unit, obtain the expert module corresponding to the first expert module from the first storage unit, and use the expert module corresponding to the first expert module to process the first word.

[0582] As another implementation, when the first storage unit does not contain the second expert module corresponding to the first expert module, the third thread may obtain the expert module corresponding to the first expert module from the second storage unit, and use the expert module corresponding to the first expert module to process the first word. The third thread may be used to indicate the processor thread. The first thread may be used to indicate the accelerator thread.

[0583] Since the first thread can be used to indicate the accelerator thread, the third thread can be used to indicate the processor thread, the first storage unit can be used to indicate the accelerator memory, the second storage unit can be used to indicate the processor memory, the third speed can be used to indicate the speed at which the third thread accesses the second storage unit, and the fourth speed can be used to indicate the speed at which the third thread accesses the first storage unit, the third access speed is greater than the fourth access speed. When the first thread determines that the first storage unit does not contain the second expert module corresponding to the first expert module, the third thread is used to obtain the expert module corresponding to the first expert module from the second storage unit, and the expert module corresponding to the first expert module is used to process the first word, without the first thread first loading the expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtaining the expert module corresponding to the first expert module from the first storage unit, and then using the expert module corresponding to the first expert module to process the first word, thereby reducing the I / O delay between the second storage unit and the first storage unit and improving the reasoning efficiency of the first model.

[0584] When the first storage unit satisfies the first condition, how to adjust the second expert module stored in the first storage unit can be achieved in the following manner.

[0585] As an implementation, the fourth thread may eliminate the second expert module from the first set stored in the first storage unit based on the cache elimination strategy when the first storage unit satisfies the first condition. The first condition may be used to trigger cache elimination. The fourth thread may be executed in parallel with the first thread and the second thread.

[0586] Since the fourth thread can eliminate the second expert module from the first set stored in the first storage unit based on the cache elimination strategy when the first storage unit meets the first condition, the fourth thread can be executed in parallel with the first thread and the second thread, thereby improving the cache hit rate and resource utilization when the storage capacity of the first storage unit is limited.

[0587] The first condition may include at least one of the following: the first capacity reaches the second threshold, the first number reaches the third threshold, the first usage rate reaches the fourth threshold, or the first hit rate is less than or equal to the fifth threshold. The first capacity can be used to indicate the total storage size of the first set. The second threshold can be determined based on the storage capacity of the first storage unit. The first number can be used to indicate the total number of second expert modules in the first set. The third threshold can be determined based on the storage capacity of the first storage unit and the storage size of the second expert module. The first usage rate can be used to indicate the ratio between the first capacity and the storage capacity of the first storage unit. The first hit rate can be used to indicate the ratio between the second number and the third number. The second number can be used to indicate the number of cache hits corresponding to the first storage unit. The third number can be used to indicate the total number of accesses corresponding to the first storage unit. And / or

[0588] The cache elimination policy may include at least one of the following: a first-in-first-out policy, a least recently used policy, a least frequently used policy, a most recently used policy, a random elimination policy, or a low interactive reference interval policy.

[0589] It should be noted that, for the description of how the fourth thread eliminates the second expert module from the first set stored in the first storage unit based on the cache elimination strategy, please refer to the corresponding part above, which will not be repeated here.

[0590] As an implementation, the first storage unit may be used to indicate an accelerator memory, and the second storage unit may be used to indicate a processor memory or a storage device.

[0591] As another implementation, the first storage unit may be used to indicate a processor memory, and the second storage unit may be used to indicate a storage device.

[0592] Since the second storage unit can be used to store multiple expert modules included in the first model, and the first storage unit can be used to store the expert modules of the activated experts, the cooperation between the second storage unit and the first storage unit alleviates the limited memory capacity of the end-side device, making it possible for the end-side device to deploy the first model.

[0593] As an implementation, accelerator memory may be used to refer to graphics processor memory, neural network processor memory, or tensor processor memory.

[0594] As an implementation, the third thread may be a processor thread. In the case where the first storage unit may be used to indicate the accelerator memory, the first thread and the second thread may be accelerator threads. The fourth thread may be an accelerator thread. In the case where the first storage unit may be used to indicate the processor memory, the first thread, the second thread, and the fourth thread may be processor threads.

[0595] It should be noted that the module names involved in the embodiments of the present application can be defined as other names as long as the functions of each module can be achieved, and there is no specific restriction on the names of the modules.

[0596] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of the present application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0597] The data processing method of the embodiment of the present application has been described above, and the device for executing the above method provided by the embodiment of the present application is described below. Those skilled in the art can understand that the method and the device can be combined and referenced with each other, and the relevant device provided by the embodiment of the present application can execute the steps in the above list sorting method.

[0598] Fig.19 This is a structural block diagram of a data processing device provided in an embodiment of the present application.

[0599] like Fig.19 As shown, the data processing device 1900 may include an acquisition module 1910 and a processing module 1920 .

[0600] The acquisition module 1910 is used for the first thread to acquire the second expert module corresponding to the first expert module from the first storage unit when the first thread executes the second stage process using the first model and when the first thread has the second expert module corresponding to the first expert module in the first storage unit. The first expert module is determined by the first thread from the plurality of expert modules included in the first model according to the first word corresponding to the first request.

[0601] The processing module 1920 is used for the first thread to process the first word by using the second expert module corresponding to the first expert module.

[0602] According to an embodiment of the present application, the first set stored in the first storage unit may be determined by the second thread according to the first request when the first thread executes the first stage process using the first model. The first set may include multiple second expert modules. The first stage process may be used to indicate that the first request is processed using the first model to obtain the second word element. The second word element may be used to indicate the first output word element corresponding to the first request. The second stage process may be used to indicate that the inference result is obtained according to the first request and the second word element using the first model.

[0603] The data processing method provided in the embodiment of the present application can be applied to a terminal device with a communication function. The specific device form of the terminal device can refer to the above related description, which will not be repeated here.

[0604] The embodiment of the present application provides an end-side device, which may include a processor and a memory. The memory stores computer-executable instructions. The processor executes the computer-executable instructions stored in the memory, so that the end-side device executes the data processing method in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments, and will not be repeated here. The processor may include at least one of a central processing unit, a graphics processing unit, a neural network processor, or a tensor processor.

[0605] The embodiment of the present application provides a chip or a chip system. The chip or chip system includes at least one processor and a communication interface. The communication interface and the at least one processor are interconnected by a line. At least one processor is used to run a computer program or instruction to execute the data processing method in the above embodiment. Its implementation principle and technical effect are similar to those of the above-mentioned related embodiments and are not repeated here. At least one processor may include at least one of a central processing unit, a graphics processing unit, a neural network processor, or a tensor processor.

[0606] The embodiments of the present application also provide a computer-readable storage medium. The computer-readable storage medium stores a computer program. The above method is implemented when the computer program is executed by the processor. The method described in the above embodiment can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. If implemented in software, the function can be stored as one or more instructions or codes on a computer-readable medium or transmitted on a computer-readable medium. Computer-readable media can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium that can be accessed by a computer.

[0607] In one possible implementation, a computer-readable medium may include a random access memory (RAM), a read-only memory (ROM), a compact disc read-only memory (CD-ROM) or other optical disk storage, a magnetic disk storage or other magnetic storage device, or any other medium that is intended to carry or store the required program code in the form of instructions or data structures and can be accessed by a computer. Moreover, any connection is appropriately referred to as a computer-readable medium. For example, if the software is transmitted from a website, server or other remote source using a coaxial cable, a fiber optic cable, a twisted pair, a digital subscriber line (DSL) or wireless technology (e.g., infrared, radio or microwave), the coaxial cable, fiber optic cable, twisted pair, DSL or wireless technology such as infrared, radio or microwave is included in the definition of the medium. Disks and optical disks as used herein include optical disks, laser disks, digital versatile discs (DVD), floppy disks or Blu-ray discs. Disks typically reproduce data magnetically, while optical disks reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0608] An embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed, the computer executes the above method.

[0609] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable data processing device generate a device for implementing the functions specified in one process or multiple processes in the flowchart and / or one box or multiple boxes in the block diagram.

[0610] The above specific implementation methods further explain in detail the purpose, technical solutions and beneficial effects of the embodiments of the present application. It should be understood that the above are only specific implementation methods of the embodiments of the present application and are not used to limit the protection scope of the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made on the basis of the technical solutions of the embodiments of the present application should be included in the protection scope of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: When the first thread executes the second stage process using the first model, The first thread obtains the second expert module corresponding to the first expert module from the first storage unit when the second expert module corresponding to the first expert module exists in the first storage unit, wherein the first expert module is determined by the first thread from a plurality of expert modules included in the first model according to the first word corresponding to the first request; The first thread processes the first word using a second expert module corresponding to the first expert module; Among them, the first set stored in the first storage unit is determined by the second thread according to the first request when the first thread executes the first stage process using the first model, the first set includes multiple second expert modules, the first stage process is used to use the first model to process the first request to obtain a second word element, the second word element is used to indicate the first output word element corresponding to the first request, and the second stage process is used to use the first model to obtain an inference result according to the first request and the second word element.

2. The method according to claim 1, characterized in that Also includes: When the first thread executes the first stage process using the first model, The second thread determines a plurality of third expert modules from the plurality of expert modules according to the first request; The second thread determines, from a second set, second expert modules corresponding to each of the plurality of third expert modules, wherein the second set is stored in the first storage unit, the second set includes a plurality of the second expert modules, and the second set is loaded from the second storage unit to the first storage unit by the first thread; The second thread retains the second expert modules in the second set corresponding to each of the plurality of third expert modules to obtain the first set.

3. The method according to claim 2, characterized in that Also includes: The first thread determines a third word according to the first request; The first thread determines at least one second expert module corresponding to the third word-gram from the plurality of expert modules; The first thread loads at least one second expert module corresponding to the third word-gram from the second storage unit to the first storage unit to obtain the second set.

4. The method according to claim 2 or 3, characterized in that: The second thread determines, according to the first request, a plurality of third expert modules from the plurality of expert modules, including: The second thread determines the multiple third expert modules from the multiple expert modules based on the first request using a second model, wherein the second model is trained using the second request and multiple first scores corresponding to the second request, and the first score is used to indicate the activation score of the expert module.

5. The method according to any one of claims 1 to 3, characterized in that: Also includes: The first thread acquires the second expert module corresponding to the fourth expert module from the first storage unit when the first storage unit does not contain the second expert module corresponding to the first expert module and the first storage unit contains the second expert module corresponding to the fourth expert module; The first thread processes the first word using a second expert module corresponding to the fourth expert module; Among them, the first expert module has a corresponding first weight, the first weight is used to indicate the activation probability of the first expert module, the fourth expert module has a corresponding second weight, the second weight is used to indicate the activation probability of the fourth expert module, and the absolute value of the difference between the first weight and the second weight is less than or equal to the first threshold.

6. The method according to claim 5, characterized in that Also includes: The first thread calls a gating module included in the first model, and uses the gating module to obtain third weights corresponding to each of the plurality of expert modules according to a first word corresponding to the first request; The first thread determines the first weight according to third weights corresponding to each of the plurality of expert modules; The first thread determines the second weight from third weights corresponding to each of the plurality of expert modules according to the first weight and the first threshold.

7. The method according to claim 5, characterized in that The fourth expert module is the second expert module in the first set corresponding to the first request; or The first set further includes a plurality of second expert modules corresponding to a third request, and the fourth expert module is a second expert module in the first set corresponding to the third request, wherein the third request is different from the first request.

8. The method according to claim 5, characterized in that Also includes: The first thread loads an expert module corresponding to the first expert module from the second storage unit to the first storage unit, obtains an expert module corresponding to the first expert module from the first storage unit, and processes the first word by using the expert module corresponding to the first expert module, if the first storage unit does not contain a second expert module corresponding to the first expert module and the first storage unit does not contain a second expert module corresponding to the fourth expert module; or When the first storage unit does not contain a second expert module corresponding to the first expert module and the first storage unit does not contain a second expert module corresponding to the fourth expert module, the third thread obtains the expert module corresponding to the first expert module from the second storage unit, and uses the expert module corresponding to the first expert module to process the first word, wherein the third thread is used to indicate a processor thread, and the first thread is used to indicate an accelerator thread.

9. The method according to any one of claims 1 to 3, characterized in that: Also includes: The first thread loads an expert module corresponding to the first expert module from the second storage unit to the first storage unit if the first storage unit does not contain a second expert module corresponding to the first expert module, obtains an expert module corresponding to the first expert module from the first storage unit, and processes the first word by using the expert module corresponding to the first expert module; or When the first storage unit does not contain a second expert module corresponding to the first expert module, the third thread obtains the expert module corresponding to the first expert module from the second storage unit, and uses the expert module corresponding to the first expert module to process the first word, wherein the third thread is used to indicate a processor thread, and the first thread is used to indicate an accelerator thread.

10. The method according to any one of claims 1 to 3, characterized in that: Also includes: The fourth thread eliminates the second expert module from the first set based on the cache elimination strategy when the first storage satisfies the first condition, wherein the first condition is used to trigger cache elimination, and the fourth thread is executed in parallel with the first thread and the second thread.

11. The method according to claim 10, characterized in that The first condition includes at least one of the following: the first capacity reaches a second threshold, the first number reaches a third threshold, the first usage rate reaches a fourth threshold, or the first hit rate is less than or equal to a fifth threshold, the first capacity is used to indicate the total storage size of the first set, the second threshold is determined according to the storage capacity of the first storage unit, the first number is used to indicate the total number of second expert modules in the first set, the third threshold is determined according to the storage capacity of the first storage unit and the storage size of the second expert modules, the first usage rate is used to indicate the ratio between the first capacity and the storage capacity of the first storage unit, the first hit rate is used to indicate the ratio between the second number and the third number, the second number is used to indicate the number of cache hits corresponding to the first storage unit, and the third number is used to indicate the total number of accesses corresponding to the first storage unit; and / or The cache elimination strategy includes at least one of the following: a first-in-first-out strategy, a least recently used strategy, a least frequently used strategy, a most recently used strategy, a random elimination strategy, or a low interactive reference interval strategy.

12. The method according to any one of claims 1 to 3, characterized in that The first storage unit is used to indicate an accelerator memory, and the second storage unit is used to indicate a processor memory or a storage device; or The first storage unit is used to indicate the processor memory, and the second storage unit is used to indicate the storage device.

13. The method according to claim 12, characterized in that The accelerator memory is used to indicate a graphics processor memory, a neural network processor memory, or a tensor processor memory.

14. A terminal side device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the terminal-side device executes the method as described in any one of claims 1-13.

15. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. A chip system, characterized in that: The method comprises at least one processor and a communication interface, wherein the communication interface and the at least one processor are interconnected via a line, and the at least one processor is used to run a computer program or instruction to execute the method according to any one of claims 1 to 13.

17. A computer program product, characterized in that It includes a computer program, which, when executed, enables the end-side device to execute the method as described in any one of claims 1 to 13.

Images