Method, computing device, medium, and program product for performing calculations on a neural network model structure

Abstract

The present invention relates to a method, computing device, medium, and program product for performing calculations on a neural network model structure. The method includes: adjusting the execution order of computing tasks on the neural network model structure so that a constructed fusion kernel function is used to perform computing tasks on a first network module and computing tasks on a second network module in parallel, wherein the first and second network modules are included in the neural network model structure and have no data dependency relationship with each other; and selecting, based on computing core identifiers, some computing cores in a corresponding artificial intelligence chip to respectively perform computing tasks on the first network module and computing tasks on the second network module. The present invention can fully utilize the computing cores of the artificial intelligence chip.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of artificial intelligence, and more specifically to a method, computing device, computer-readable storage medium, and computer program product for performing calculations on a neural network model structure. Background Art

[0002] Traditional solutions for performing computations on neural network model structures typically require sequential processing of the multiple network modules included in the neural network model. For example, this neural network model is a Mixture of Experts (MoE) structure based on a Large Language Model (LLM). It should be understood that the Feed-Forward Network (FFN) within the MoE network structure is one of the model's core computational modules. The FFN modules have a unique interconnected relationship, and their computational time accounts for a significant portion of the MoE's overall computational time. Specifically, the computation first requires computing the Router TopK module, then the Activated Routed Expert module, and finally the Shared Expert module, resulting in a significant computational overhead.

[0003] Traditional solutions for performing calculations on neural network model structures typically use different kernel functions to calculate the Router TopK, Activated Routed Expert, and Shared Expert modules. However, when calculating the Router TopK module, the relatively small amount of computational tasks involved prevents full utilization of all the AI ​​chip's computing cores. Furthermore, when calculating the Shared Expert module, the relatively small number of Shared Experts also prevents full utilization of the AI ​​chip's computing cores, resulting in inefficient task division and execution within the AI ​​chip.

[0004] In summary, the traditional solution for performing calculations on neural network model structures has the following shortcomings: it is difficult to fully utilize the computing cores of artificial intelligence chips when performing computing tasks on neural network models. Summary of the Invention

[0005] The present invention provides a method, computing device, computer-readable storage medium, and computer program product for performing calculations on a neural network model structure, which can fully utilize the computing core of an artificial intelligence chip.

[0006] According to a first aspect of the present invention, a method for performing calculations on a neural network model structure is provided. The method comprises: adjusting the execution order of calculation tasks on the neural network model structure so that a constructed fusion kernel function is used to perform calculation tasks on a first network module and calculation tasks on a second network module in parallel, wherein the first network module and the second network module are included in the neural network model structure and have no data dependency relationship with each other; and selecting, based on a calculation core identifier, some calculation cores in a corresponding artificial intelligence chip to respectively perform the calculation tasks on the first network module and the calculation tasks on the second network module.

[0007] In some embodiments, the method for performing calculations on a neural network model structure also includes: calculating the ratio between the amount of computing tasks on the first network module and the amount of computing tasks on the second network module, so as to determine the number of computing cores required for the computing tasks on the first network module based on the ratio; and constructing a fusion kernel function, which is configured to: select between the computing tasks on the first network module and the computing tasks on the second network module based on at least the current computing core identifier.

[0008] In some embodiments, the fusion kernel function is configured to: at least based on the current computing core identifier, select between the computing task regarding the first network module and the computing task regarding the second network module, including: in response to determining that the current computing core identifier is less than the split task identifier, select the computing task regarding the first network module; and in response to determining that the current computing core identifier is greater than or equal to the split task identifier, select the computing task regarding the second network module.

[0009] In some embodiments, determining the number of computing cores required for the computing task regarding the first network module based on the ratio includes: determining the number of computing cores required for the computing task regarding the first network module based on the ratio, thereby determining a split task identifier, wherein the split task identifier indicates the location division status of the computing task when it is executed on the computing core of the corresponding artificial intelligence chip.

[0010] In some embodiments, constructing a fusion kernel function includes: modifying the first original kernel function used to calculate the first network module into a first device function, and making the number of computing cores of the first device function equal to the split task identifier, and the first original kernel function is a global function; and modifying the second original kernel function used to calculate the second network module into a second device function, and making the number of computing cores of the second device function equal to the difference between the computing core identifier of the artificial intelligence chip and the split task identifier, and making the task identifier of the second device function equal to the difference between the computing core identifier of the artificial intelligence chip and the split task identifier, and the second original kernel function is a global function.

[0011] In some embodiments, the neural network model structure is a hybrid expert structure of a large language model, the first network module is a pre-routing pre-positioning expert module, and the second network module is a shared expert module.

[0012] In some embodiments, constructing the fusion kernel function includes configuring the following parameters for the fusion kernel function: output hidden data, activation expert list, input hidden data, and segmentation task identifier.

[0013] In some embodiments, selecting the computing task regarding the first network module includes: selecting to call a first device function; and selecting the computing task regarding the second network module includes: selecting to call a second device function.

[0014] According to a second aspect of the present invention, a computing device is further provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the computing device to perform the method of the first aspect of the present invention.

[0015] According to a third aspect of the present invention, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and when the computer program is executed by a machine, the method according to the first aspect of the present invention is executed.

[0016] According to a fourth aspect of the present invention, there is further provided a computer program product, comprising a computer program, wherein when the computer program is executed by a machine, the method of the first aspect of the present invention is performed.

[0017] The present invention enables the fusion kernel function to allocate the computing cores of the artificial intelligence chip to the computing tasks of the first network module and the second network module, so that the computing cores are partitioned into different computing tasks related to different network models, thereby fully utilizing the computing cores of the artificial intelligence chip. Therefore, the present invention can fully utilize the computing cores of the artificial intelligence chip.

[0018] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other features, advantages and aspects of the embodiments of the present invention will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements.

[0020] Figure 1 A schematic diagram schematically shows a typical neural network model structure.

[0021] Figure 2 A schematic diagram schematically shows a conventional method for performing calculations on a neural network model structure.

[0022] Figure 3 A schematic diagram of a computing device implementing a method for performing calculations on a neural network model structure according to an embodiment of the present invention is schematically shown.

[0023] Figure 4 A flowchart of a method for performing calculations on a neural network model structure according to an embodiment of the present invention is shown.

[0024] Figure 5 A schematic diagram of a fusion kernel function and a graphics processor according to an embodiment of the present invention is shown.

[0025] Figure 6 A flowchart of a method for constructing a fusion kernel function according to an embodiment of the present invention is shown.

[0026] Figure 7 A schematic diagram of a code for constructing a fusion kernel function according to an embodiment of the present invention is schematically shown.

[0027] In the various drawings, the same or corresponding reference numerals denote the same or corresponding parts. DETAILED DESCRIPTION

[0028] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0029] As used herein, the term "including" and its variations represent open inclusion, i.e., "including but not limited to." The term "based on" means "based at least in part on." The terms "one example embodiment" and "an embodiment" mean "at least one example embodiment." The term "another embodiment" means "at least one additional embodiment." The terms "first," "second," and so on may refer to different or identical objects.

[0030] As described above, Figure 1 A schematic diagram of a typical neural network model structure 100 is shown. This neural network model structure 100 is, for example, a feed-forward network (FFN) model structure within a Mix of Experts (MoE) model structure within a Large Language Model (LLM). Specifically, when performing computational tasks associated with the FFN model structure within the MoE, multiple network modules must be processed sequentially. For example, the pre-routing pre-positioning (i.e., Router TopK) module 120 must first be computed. This module selects eight experts from 256 experts based on input 110. The Activated Routed Expert module then performs computations for the eight selected activated experts. Furthermore, the Shared Expert module must also be computed. The computational results of the Shared Expert network module and the Activated Routed Expert module are superimposed to generate output 130.

[0031] Figure 2 A schematic diagram of a conventional method 200 for performing calculations on a neural network model structure is shown. Studies have shown that in conventional methods for performing calculations on a neural network model structure, different kernel functions are usually used to sequentially calculate the Router TopK, Activated Routed Expert, and Shared Expert modules. For example, Figure 2 As shown in the figure, kernel function 1 is first used to perform calculations on the pre-routing pre-positioning expert module; kernel function 2 is then used to perform calculations on the routing activation expert module; and finally, kernel function 3 is used to perform calculations on the shared expert module. Table 1 below schematically illustrates the number of shared experts, the number of activated experts, and the total number of routing experts for several typical neural network model structures (e.g., DeepSeek V3 / R1, QWen3, and Llama4).

[0032] Table 1

[0033]

[0034] When performing calculations on the Router TopK module, the Router TopK module is responsible for selecting the K experts with the highest weights from all experts. The computational task of this step is relatively simple, and the computational characteristics are memory-bound. Therefore, the GPGPU computing cores are not fully utilized, resulting in low GPGPU utilization. When calculating the Activated Routed Expert module, multiple Routed Experts are activated simultaneously. Therefore, all GPGPU computing cores can be fully utilized, resulting in relatively high GPGPU utilization. When calculating the Shared Expert module, usually only a small number of Shared Experts are activated, and the GPGPU computing cores are not fully utilized. Therefore, the overall GPGPU utilization is relatively low.

[0035] In summary, the traditional solution for performing calculations on neural network model structures has the following shortcomings: it is difficult to fully utilize the computing cores of artificial intelligence chips when performing computing tasks on neural network models.

[0036] To at least partially address one or more of the aforementioned issues and other potential problems, exemplary embodiments of the present invention propose a solution for performing computations related to a neural network model structure. This solution adjusts the execution order of computational tasks related to the neural network model structure so that a fused kernel function executes computational tasks for a first network module and a second network module in parallel, without data dependencies between them. Furthermore, based on computational core identifiers, a subset of computational cores is selected to separately execute computational tasks for the first and second network modules. This allows the fused kernel function to allocate computational cores of an artificial intelligence chip to computational tasks for the first and second network modules, partitioning the computational cores into different computational tasks related to different network models and thereby fully utilizing the computational cores of the artificial intelligence chip. Therefore, the present invention can fully utilize the computational cores of an artificial intelligence chip. It should be understood that the artificial intelligence chip may be, for example, but not limited to, a graphics processing unit (GPU), a general-purpose graphics processing unit (GPGPU), or a tensor processing unit (TPU).

[0037] Figure 3 Schematically shows a schematic diagram of a computing device 300 for implementing a method for performing calculations on a neural network model structure according to an embodiment of the present invention. Figure 3As shown, the computing device 300 may have one or more processing units, including dedicated processing units such as a graphics processing unit (GPU), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a general-purpose computing on graphics processing unit (GPGPU), as well as general-purpose processing units such as a CPU. The computing device 300 also includes at least: a neural network model structure computing task execution order adjustment unit 302 and a computing core selection unit 304. It should be understood that the neural network model structure computing task execution order adjustment unit 302 and the computing core selection unit 304 can be software modules, which, for example, run on one or more processing units configured by the computing device 300.

[0038] Regarding the neural network model structure computing task execution order adjustment unit 302, it is used to adjust the execution order of the computing tasks regarding the neural network model structure so that the constructed fusion kernel function is used to execute the computing tasks regarding the first network module and the computing tasks regarding the second network module in parallel. The first network module and the second network module are included in the neural network model structure, and there is no data dependency relationship between them.

[0039] Regarding the computing core selection unit 304, it is used to select some computing cores in the corresponding artificial intelligence chip according to the computing core identifier to respectively execute the computing tasks regarding the first network module and the computing tasks regarding the second network module.

[0040] The following will be combined Figure 4 and Figure 5 A method 400 for performing calculations on a neural network model structure according to an embodiment of the present invention is described. Figure 4 A flowchart of a method 400 for performing calculations on a neural network model structure according to an embodiment of the present invention is shown. Figure 5 FIG4 shows a schematic diagram of a fusion kernel function and a graphics processor according to an embodiment of the present invention. It should be understood that the method 400 can be used, for example, in Figure 3 The method 400 is executed at the described computing device 300. The method 400 may also include additional actions not shown and / or may omit actions shown, and the scope of the present invention is not limited in this respect.

[0041] At step 402, the computing device 300 adjusts the execution order of computing tasks regarding the neural network model structure so that the constructed fusion kernel function is used to execute computing tasks regarding the first network module and computing tasks regarding the second network module in parallel, wherein the first network module and the second network module are included in the neural network model structure and there is no data dependency relationship between them.

[0042] Regarding the neural network model structure, for example, it includes at least: a first network module and a second network module. There is no data dependency between the first network module and the second network module. It should be understood that the neural network model structure may also include: other network modules in addition to the first network module and the second network module, for example, a third network module. In some embodiments, the neural network model structure is a hybrid expert structure of a large language model, where the first network module is a pre-routing pre-positioning expert module and the second network module is a shared expert module.

[0043] For example, on the computation graph, there's no data dependency between the computational tasks of the pre-routing pre-positioning expert module and those of the shared expert module. Therefore, computing device 300 can adjust the execution order of the computational tasks (for example, by adjusting the computation graph) so that the constructed fusion kernel function executes the computational tasks for the pre-routing pre-positioning expert module and the shared expert module in parallel. This improves the utilization of the AI ​​chip while maintaining the original computational logic and accuracy.

[0044] like Figure 5As shown, fused kernel function 510 is formed by fusing two kernel functions: one for executing the computational task of the first network module and the other for executing the computational task of the second network module. Specifically, fused kernel function 510 is configured to include a first subtask execution unit 514 and a second subtask execution unit 516. The first subtask execution unit is used to execute the computational task associated with the first network module, while the second subtask execution unit 516 is used to execute the computational task associated with the second network module. In some embodiments, computing device 300 adjusts the execution order of computational tasks by adjusting the computation graph, such that the first original kernel function for computing the Router TopK module and the third original kernel function for computing the Shared Expert module are fused into fused kernel function 510. In this way, the first subtask execution unit 514 of fused kernel function 510 executes the computational task associated with the Router TopK module, while the second subtask execution unit 516 executes the computational task associated with the Shared Expert module. After the fusion kernel function 510 , the second kernel function 520 performs the computation task of activating the Routed Expert module.

[0045] In some embodiments, the fusion kernel function is constructed by the following method: modifying the first original kernel function used to calculate the first network module into a first device function, and making the number of computing cores of the first device function equal to the split task identifier, and the first original kernel function is a global function; and modifying the second original kernel function used to calculate the second network module into a second device function, and making the number of computing cores of the second device function equal to the difference between the computing core identifier of the artificial intelligence chip and the split task identifier, and making the task identifier of the second device function equal to the difference between the computing core identifier of the artificial intelligence chip and the split task identifier, and the second original kernel function is a global function.

[0046] At step 404 , the computing device 300 selects some computing cores in the corresponding artificial intelligence chip according to the computing core identifiers to respectively execute computing tasks related to the first network module and computing tasks related to the second network module.

[0047] For example, taking the artificial intelligence chip as a graphics processor as an example, the subtask selection unit 512 configured at the entrance of the fusion kernel function 510, at least based on the computing core identifier, selects the first part of the computing core 532 in the graphics processor 530 corresponding to the fusion kernel function 510 to perform the computing task on the first network module (for example, the computing task on the Router TopK module executed by the first subtask execution unit 514); and selects the second part of the computing core 534 in the corresponding graphics processor 530 to perform the computing task on the second network module (that is, the computing task on the Shared Expert module executed by the second subtask execution unit 516).

[0048] Table 2 below illustrates a conventional code for allocating compute cores for a computation task related to a first network module (e.g., Router Top K). The first primitive kernel function used to perform computations related to the first network module (e.g., Router Top K) is a global function, such as indicated by "_global__ void router(ActivatedExpertListae, InputHiden ih)" in Table 2. Furthermore, the code "computeCoreNum = gpu_compute_core_num()" sets the number of compute cores to the compute core ID of the artificial intelligence chip (e.g., graphics processor). Using a graphics processor as an example, the number of compute cores is set to 16. Furthermore, the code "taskID = gpu_compute_core_id()" sets the task ID to the compute core ID of the graphics processor. The current compute core task is then obtained, and the computation task related to the first network module (e.g., Router Top K) is executed on all 16 compute cores, starting with the corresponding task ID. It can be seen that although the computing task of the first network module (for example, Router TopK) is relatively small, the computing task of the first network module (for example, Router TopK) is assigned to the 16 computing cores of the image processor, which makes the computing task assigned to each computing core even smaller, and thus the computing power of the image processor is not fully utilized, resulting in low utilization and computing efficiency of the image processor.

[0049] Table 2

[0050]

[0051] Table 3 below schematically shows the code for the traditional allocation of computing cores for computing tasks related to the second network module (e.g., Shared Expert). The second original kernel function used to execute the computing tasks related to the second network module (e.g., Shared Expert) is a global function, for example, as indicated by "_global__ void sharedExpert(OutputHidden oh,InputHiden ih)" in Table 2. In addition, the code "computCoreNum = gpu_compute_core_num()" is used to make the number of computing cores equal to the computing core identifier of the image processor. For example, the number of computing cores is made equal to 16. The code "taskID = gpu_compute_core_id()" is used to make the task identifier equal to the computing core identifier of the image processor. Afterwards, the current computing core task is obtained, and the computing tasks related to the second network module (e.g., Shared Expert) are executed on the 16 computing cores starting from the corresponding task identifier.

[0052] Table 3

[0053]

[0054] It can be seen from the above code that in the traditional method for performing calculations on the neural network model structure, if the computing tasks on the first network module or the computing tasks on the second network module are run separately on the corresponding artificial intelligence chip (for example, a graphics processor), the amount of computing tasks performed on each of the 16 computing cores of the artificial intelligence chip is small, making it difficult to fully utilize the 16 computing cores of the artificial intelligence chip.

[0055] Table 4 below exemplifies the code for allocating computing cores for computing tasks related to a first network module (e.g., RouterTopK) according to an embodiment of the present invention. As shown in Table 4, the fused kernel function used to compute the computing tasks for the first network module (e.g., RouterTopK) is configured as a first device function, as indicated, for example, by "__device__void router(ActivatedExpertList ae, InputHiden ih, int taskSplitID)" in Table 4. Regarding the method for configuring as a first device function, for example, it includes: modifying the first original kernel function used to compute the first network module (as shown in Table 2) from a global function to a first device function. It should be understood that global functions cannot be called by other kernel functions. Modifying the first original kernel function used to compute the first network module from a global function to a first device function with a device function attribute enables the calling of the first device function.

[0056] As shown in Table 4, the code "computeCoreNum = taskSplitID" is used to make the number of computing cores equal to the corresponding split task identifier. Among them, taskSplitID represents the corresponding split task identifier, which is the input parameter of the first device function. For example, taskSplitID=8 makes the number of computing cores equal to 8. In addition, the code "taskID = gpu_compute_core_id()" in Table 4 makes the task identifier equal to the computing core identifier of the image processor. Afterwards, the current computing core task is obtained, and starting from the corresponding task identifiers of the 8 computing cores, the computing tasks of the pre-routing expert module are executed. Figure 5 As shown, the gpu_compute_core_id() of the computing core 5 is 5, and its taskID is 5, then the computing core 5 is used to execute the computing task with taskID 5 on the first network module.

[0057] Table 4

[0058]

[0059] Table 5 below schematically illustrates code for allocating computing cores for a second network module (e.g., SharedExpert) according to an embodiment of the present invention. As shown in Table 5, the fused kernel function used for computing the second network module (e.g., SharedExpert) is configured as a second device function, for example, as indicated by "__device__ void router(ActivatedExpertList ae, InputHiden ih, int taskSplitID)" in Table 5. Regarding the method for configuring it as a second device function, for example, it includes: modifying the second original kernel function used for computing the second network module (e.g., SharedExpert) (as shown in Table 3) from a global function to a second device function.

[0060] In addition, as shown in Table 5, the relationship between the number of computing cores (computeCoreNum) of the second device function, the computing core identifier of the corresponding graphics processor (gpu_compute_core_num()), the split task identifier (taskSplitID), and the task identifier (taskID) is configured. Specifically, for example, through the code "computeCoreNum = gpu_compute_core_num() - taskSplitID", the number of computing cores is equal to the number of computing cores of the graphics processor minus the corresponding split task identifier; and through the code "taskID = gpu_compute_core_id() - taskSplitID", the task identifier is equal to the computing core identifier of the corresponding graphics processor minus the corresponding split task identifier. For example, taskSplitID=8, as Figure 5 As shown, the gpu_compute_core_id() of the computing core 12 is 12, and its taskID is 12 - 8 = 4, so the computing core 12 is used to execute the computing task with taskID 4 on the second network module.

[0061] Through the above-mentioned means of the present invention, two different computing tasks regarding different network modules are assigned to different partitions of an artificial intelligence chip (e.g., a graphics processor) (e.g., computing cores 0 to 8 are used to perform computing tasks regarding the first network module, and computing cores 9 to 15 are used to perform computing tasks regarding the second network module), so that the calculations regarding the first network module and the calculations regarding the second network module can fully utilize the computing cores of the processor, and the spatial partitions of different computing cores can execute different computing tasks in parallel in time, thereby significantly improving computing efficiency.

[0062] Table 5

[0063]

[0064] In the above scheme, by adjusting the execution order of computing tasks related to the neural network model structure so that the fusion kernel function executes the computing tasks of the first network module and the computing tasks of the second network module in parallel, which have no data dependencies between them; and selecting some computing cores based on the computing core identifiers to respectively execute the computing tasks of the first network module and the computing tasks of the second network module, the present invention enables the fusion kernel function to allocate the computing cores of the artificial intelligence chip to the computing tasks of the first network module and the second network module, so that the computing cores are partitioned into different computing tasks related to different network models, thereby fully utilizing the computing cores of the artificial intelligence chip. Therefore, the present invention can fully utilize the computing cores of the artificial intelligence chip.

[0065] In some embodiments, the method 400 further includes a method 600 for constructing a fusion kernel function.

[0066] The following will be combined Figure 6 and Figure 7 A method 600 for constructing a fusion kernel function according to an embodiment of the present invention is described. Figure 6 A flowchart of a method 600 for constructing a fusion kernel function according to an embodiment of the present invention is shown. Figure 7 Schematic diagram of the code for constructing the fusion kernel function according to an embodiment of the present invention is shown. It should be understood that the method 600 can be used in Figure 3 The method 600 is executed at the described computing device 300. The method 600 may also include additional actions not shown and / or may omit actions shown, and the scope of the present invention is not limited in this respect.

[0067] At step 602 , the computing device 300 calculates a ratio between the amount of computing tasks for the first network module and the amount of computing tasks for the second network module, so as to determine the number of computing cores required for the computing tasks for the first network module based on the ratio.

[0068] For example, the computing device 300 calculates the number of computing cores required for computing tasks related to the network structure of the first network module. For example, the number of computing cores required may also vary depending on the size of the computing tasks of the first network module. For example, a graphing calculator has 16 computing cores. If the ratio of the computing tasks of the Router TopK module and the Shared Expert module used in the DeepSeek model is 1:1, then the number of computing cores required by the computing device 300 for the computing tasks of the first network module is 8. For example, the Router TopK module used in QWen3 has fewer computing tasks, and the ratio of the computing tasks of the Router TopK module and the Shared Expert module is 1:4. Therefore, the number of computing cores required for the Router TopK module of QWen3 is 4.

[0069] In some embodiments, determining the number of computing cores required for the computing task of the first network module based on the ratio includes: determining the number of computing cores required for the computing task of the first network module based on the ratio, and determining a split task identifier, wherein the split task identifier indicates a location division status of task execution, for example, indicating how many computing cores are used to execute the computing task of the first network module and how many computing cores are used to execute the computing task of the second network module. The split task identifier is, for example, represented as "taskSplitID".

[0070] At step 604 , the computing device 300 constructs a fusion kernel function.

[0071] Table 6 below illustrates exemplary code for constructing a fusion kernel function according to an embodiment of the present invention. As shown in Table 6, a fusion kernel function is defined, for example, as indicated by "__global__ void fuseWork(OutputHidden oh, ActivatedExpertList ae, InputHiden ih, int taskSplitID)". This fusion kernel function is configured with the following parameters: output hidden data "OutputHidden oh", activated expert list "ActivatedExpertList ae", input hidden data "InputHiden ih", and split task identifier "int taskSplitID". For example, the code "coreID = gpu_compute_core_id()" sets the current compute core ID to the image processor's compute core ID.

[0072] It should be understood that the fusion kernel function is configured to select between the computing task associated with the first network module and the computing task associated with the second network module based at least on the current computing kernel identifier. Specifically, the method for selecting between the computing task associated with the first network module and the computing task associated with the second network module includes, for example, steps 606 to 610.

[0073] At step 606 , the computing device 300 determines whether the current computing core identifier is smaller than the split task identifier.

[0074] At step 608 , if the computing device 300 determines that the current computing core identifier is smaller than the split task identifier, the computing task associated with the first network module is selected.

[0075] like Figure 7 As shown, if the computing device 300 determines that the current computing core identifier is smaller than the split task identifier, as indicated by arrow 710 , the first device function 712 is selected to be called.

[0076] For example, the code "if(coreID < taskSplitID) { router(ae, ih,taskSplitID)" in Table 6 allows, when the current computing core ID (coreID) is less than the split task ID (taskSplitID), the calculation of the first device function for the computing task of the Router TopK module is executed, for example, as indicated by "router(ae, ih, taskSplitID)" in Table 6. For another example, when the previous computing core ID (coreID) is "0" but less than the split task ID (taskSplitID is, for example, 8), the first device function for the computing task of the Router TopK module is selected.

[0077] At step 610 , if the computing device 300 determines that the current computing core identifier is greater than or equal to the split task identifier, the computing task of the second network module is selected.

[0078] like Figure 7 As shown, if the computing device 300 determines that the current computing core identifier is greater than or equal to the split task identifier, as indicated by arrow 720 , the second device function 722 is selected to be called.

[0079] For example, through the code “} else sharedExpert(oh, ih, taskSplitID);}” in Table 6, when the current computing core identifier (coreID) is greater than or equal to the split task identifier (taskSplitID), the calculation of the second device function of the computing task of the shared expert (Shared Expert) module is executed, for example, as indicated by “sharedExpert(oh, ih, taskSplitID)” in Table 6. For example, if the current computing core identifier (coreID) is “9”, which is greater than the split task identifier (taskSplitID is, for example, 8), the calculation of the second device function of the computing task of the Shared Expert module is selected.

[0080] Table 6

[0081]

[0082] In the above solution, the present invention can reuse the codes of the existing first original kernel function and the second original kernel function without affecting any network structure and completely maintaining the accuracy of the network.

[0083] The various processes and processing described above, such as methods 400 and 600, may be performed on a computing device. The computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively coupled to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor. In some embodiments, methods 400 and 600 may be implemented as a computer software program or program product tangibly embodied on a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed onto the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 400 and 600 described above may be performed.

[0084] The present invention may be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for performing various aspects of the present invention. The computer-readable storage medium may be a tangible device that can retain and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0085] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0086] These computer-readable program instructions can be provided to a central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine such that when these instructions are executed by the central processing unit of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0087] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the prescribed logical function. In some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented using a dedicated hardware-based system that performs the prescribed function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0088] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this application can be achieved. This is not a limitation herein.

[0089] The above specific embodiments do not constitute a limitation on the scope of protection of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors.

Claims

1. A method for performing calculations on a neural network model structure, characterized in that include: Constructing a fusion kernel function, the fusion kernel function being configured to: select between a computing task related to the first network module and a computing task related to the second network module based on at least a current computing kernel identifier, wherein the neural network model structure is a hybrid expert structure of a large language model, the first network module is a pre-routing pre-positioning expert module, and the second network module is a shared expert module; Adjusting the execution order of computing tasks related to the neural network model structure so that the constructed fusion kernel function is used to execute computing tasks related to a first network module and computing tasks related to a second network module in parallel, wherein the first network module and the second network module are included in the neural network model structure and have no data dependency relationship with each other; as well as According to the computing core identifier, some computing cores in different partitions are selected in the corresponding artificial intelligence chip to respectively execute computing tasks related to the first network module and computing tasks related to the second network module.

2. The method according to claim 1, characterized in that Also includes: A ratio between the amount of computing tasks for the first network module and the amount of computing tasks for the second network module is calculated, so as to determine the number of computing cores required for the computing tasks for the first network module based on the ratio.

3. The method according to claim 2, wherein The fusion kernel function is configured to select between a computing task related to the first network module and a computing task related to the second network module based on at least the current computing kernel identifier, including: In response to determining that the current computing core identifier is less than the split task identifier, selecting a computing task related to the first network module; and In response to determining that the current computing core identifier is greater than or equal to the split task identifier, a computing task related to the second network module is selected.

4. The method according to claim 3, characterized in that Determining the number of computing cores required for the computing task of the first network module based on the ratio includes: Based on the ratio, the number of computing cores required for the computing task of the first network module is determined, thereby determining a split task identifier, which indicates the location division status of the computing task when it is executed on the computing core of the corresponding artificial intelligence chip.

5. The method according to claim 2, wherein Constructing the fusion kernel function includes: Modifying a first original kernel function for calculating the first network module into a first device function, and making the number of computing cores of the first device function equal to the split task identifier, wherein the first original kernel function is a global function; as well as The second original kernel function used to calculate the second network module is modified into a second device function, and the number of computing cores of the second device function is equal to the difference between the computing core identifier of the artificial intelligence chip and the split task identifier, and the task identifier of the second device function is equal to the difference between the computing core identifier of the artificial intelligence chip and the split task identifier. The second original kernel function is a global function.

6. The method according to claim 3, characterized in that Constructing the fusion kernel function includes: Configure the following parameters for the fusion kernel function: output hidden data, activation expert list, input hidden data, and segmentation task identifier.

7. The method according to claim 5, characterized in that Selecting a computing task related to the first network module includes: selecting to call a first device function; and Selecting the computing task related to the second network module includes: selecting to call a second device function.

8. A computing device, characterized in that: include: at least one processor; as well as a memory communicatively coupled to the at least one processor; in The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a machine, performs the method according to any one of claims 1 to 7.

10. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a machine, performs the method according to any one of claims 1 to 7.

Images