Heterogeneous many-core-based large-scale hybrid expert language model reasoning method and architecture

Abstract

The embodiment of the invention discloses a heterogeneous many-core-based large-scale hybrid expert language model reasoning method and architecture. The method comprises the steps of obtaining the number of computing devices in a heterogeneous many-core architecture and the number of cores in each computing device; according to the number of the computing devices and the number of cores in each computing device, segmenting the input tensor of the expert multilayer perceptron layer to obtain a plurality of tensor segmentation results; each tensor segmentation result is distributed to each core of each computing device, parallel expert multi-layer perceptron layer computing is carried out in each core of each computing device according to the distributed tensor segmentation result, and an expert weighting result is obtained; specialist weighting results in the cores of the computing devices are stipulated and aggregated, and a large-scale mixed expert language model reasoning result based on the heterogeneous many-cores is obtained. According to the method, tensor segmentation is carried out, calculation is placed on a proper calculation equipment core, a plurality of highly optimized parallel operators can be realized, and end-to-end efficient reasoning is completed.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a large-scale hybrid expert language model reasoning method and architecture based on heterogeneous multi-core. Background Art

[0002] The large-scale mixture of experts (LLM) provides more efficient and targeted processing capabilities by integrating multiple expert networks, each with expertise in different aspects. The mixed expert model divides the input data into multiple regions, and a gating mechanism determines which expert model processes a specific input. Each expert model focuses on processing its own area of ​​expertise. The mixed expert model only activates relevant experts during the reasoning process, thereby reducing computing costs while improving performance. It is widely used in the field of natural language processing (NLP). For example, Mistral's Mixtral 8x7B adopts the MoE architecture, which effectively combines multiple experts to improve performance.

[0003] However, the main solution for implementing large-scale hybrid expert language model reasoning on multiple computing devices is through pipeline parallelism. Specifically, the existing technology performs different operations by dividing different parts of the model onto different computing devices. For example, the model is divided into multiple sub-models or layers, each sub-model or layer performs calculations on different devices, and the sub-models or layers communicate with each other through message passing based on the communication library on the device to transfer the necessary intermediate results, ultimately achieving end-to-end reasoning of large-scale hybrid expert language models on multiple computing devices.

[0004] In pipeline parallel reasoning of the prior art, the calculations performed by each computing device are different. In addition, the data between multiple computing devices are dependent. For example, the calculation of the current stage depends on the calculation results of the previous stage. Due to the limitation of data dependency, each computing device cannot fully utilize the parallelism. In addition, since the calculations in each computing device are different, the load between the computing devices is unbalanced, affecting the overall reasoning performance. Summary of the invention

[0005] The present invention provides a large-scale hybrid expert language model reasoning method and architecture based on heterogeneous multi-cores to balance the operations in multiple computing device cores, improve the parallelism of multiple computing device cores, and further improve the reasoning performance.

[0006] According to one aspect of the present invention, a large-scale hybrid expert language model reasoning method based on heterogeneous many-cores is provided, the method comprising:

[0007] Obtain the number of computing devices in the heterogeneous many-core architecture and the number of cores in each computing device;

[0008] According to the number of computing devices and the number of cores in each computing device, the input tensor of the expert multi-layer perceptron layer is divided to obtain multiple tensor division results;

[0009] Distribute each tensor segmentation result to each core of each computing device, and perform parallel expert multi-layer perceptron layer calculation in each core of each computing device according to the distributed tensor segmentation result to obtain an expert weighted result;

[0010] The expert weighted results in each core of each computing device are reduced and aggregated to obtain the reasoning results of a large-scale hybrid expert language model based on heterogeneous multi-cores.

[0011] According to another aspect of the present invention, a large-scale hybrid expert language model reasoning architecture based on heterogeneous multi-core is provided, wherein the reasoning architecture comprises a plurality of computing devices, each computing device comprises a plurality of cores, and a layer in the large-scale hybrid expert language model is arranged on each core of each computing device; wherein:

[0012] Each core of each computing device is used to obtain a target tensor segmentation result obtained by segmenting according to the number of computing devices and the number of cores in each computing device, and perform parallel expert multi-layer perceptron layer calculation according to the target tensor segmentation result to obtain an expert weighted result;

[0013] The main core in each computing device is used to reduce and aggregate the expert weighted results in each core of each computing device to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multi-cores.

[0014] According to another aspect of the present invention, a large-scale hybrid expert language model reasoning device based on heterogeneous many-cores is provided, the device comprising:

[0015] A quantity acquisition module is used to obtain the number of computing devices in the heterogeneous many-core architecture and the number of cores in each computing device;

[0016] A tensor splitting module, used for splitting the input tensor of the expert multi-layer perceptron layer according to the number of computing devices and the number of cores in each computing device, to obtain multiple tensor splitting results;

[0017] A tensor allocation module is used to allocate each tensor segmentation result to each core of each computing device, and perform parallel expert multi-layer perceptron layer calculation in each core of each computing device according to the allocated tensor segmentation result to obtain an expert weighted result;

[0018] The reduction and aggregation module is used to reduce and aggregate the expert weighted results in each core of each computing device to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multi-cores.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] at least one processor; and

[0021] a memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores described in any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores described in any embodiment of the present invention when executed.

[0024] According to another aspect of the present invention, a computer program product is provided, including a computer program, which, when executed by a processor, implements the large-scale hybrid expert language model reasoning method based on heterogeneous many-cores as described in any embodiment of the present invention.

[0025] The technical solution of the embodiment of the present invention obtains the number of computing devices in a heterogeneous multi-core architecture and the number of cores in each computing device; according to the number of computing devices and the number of cores in each computing device, the input tensor of the expert multi-layer perceptron layer is split to obtain multiple tensor split results; each tensor split result is distributed to each core of each computing device, and in each core of each computing device, parallel expert multi-layer perceptron layer calculation is performed according to the distributed tensor split results to obtain expert weighted results; the expert weighted results in each core of each computing device are reduced and aggregated to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multi-core, which solves the problem of poor performance of the prior art when using pipeline parallelism for reasoning. By splitting the tensor and placing the calculation on the appropriate computing device core, multiple highly optimized parallel operators can be implemented to complete end-to-end efficient reasoning.

[0026] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 is a flow chart of a large-scale hybrid expert language model reasoning method based on heterogeneous multi-core provided according to the first embodiment of the present invention;

[0029] Figure 2 is a flowchart of another large-scale hybrid expert language model reasoning method based on heterogeneous multi-core provided according to the first embodiment of the present invention;

[0030] Figure 3 This is a rendering of a large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores provided according to the first embodiment of the present invention;

[0031] Figure 4 is a schematic diagram of a large-scale hybrid expert language model reasoning architecture based on heterogeneous many-cores provided according to the second embodiment of the present invention;

[0032] Figure 5 is a schematic diagram of the structure of a large-scale hybrid expert language model reasoning device based on heterogeneous multi-core provided according to Embodiment 3 of the present invention;

[0033] Figure 6 It is a structural schematic diagram of an electronic device for implementing the large-scale hybrid expert language model reasoning method based on heterogeneous many-cores according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] Embodiment 1

[0037] Figure 1 This is a flowchart of a large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores provided according to the first embodiment of the present invention. This embodiment is applicable to the case of performing large-scale hybrid expert language model parallel reasoning on multiple computing devices with multi-cores. The method can be executed by a large-scale hybrid expert language model reasoning device based on heterogeneous multi-cores. The large-scale hybrid expert language model reasoning device based on heterogeneous multi-cores can be implemented in the form of hardware and / or software. The large-scale hybrid expert language model reasoning device based on heterogeneous multi-cores can be configured in an electronic device such as a computer. Figure 1 As shown, the method includes:

[0038] Step 110: Obtain the number of computing devices in the heterogeneous many-core architecture and the number of cores in each computing device.

[0039] The technical solution of the embodiment of the present invention is implemented based on a heterogeneous many-core architecture. The heterogeneous many-core architecture can be understood as the existence of multiple computing devices in the architecture for asynchronous parallel computing, and multiple processor cores (such as CPU cores or GPU cores, etc.) in each computing device, and asynchronous parallel execution can also be adopted between each core.

[0040] In the embodiment of the present invention, in order to balance the computing tasks, the number of computing devices and the number of cores in each computing device may be determined so as to distribute the tasks according to the number of units that can perform computing.

[0041] Step 120: Split the input tensor of the expert multi-layer perceptron layer according to the number of computing devices and the number of cores in each computing device to obtain multiple tensor splitting results.

[0042] In an embodiment of the present invention, a large-scale hybrid expert language model includes: an input embedding vector layer, a multi-head attention layer, a gated matrix multiplication, an activation function and normalization layer, an expert multi-layer perceptron layer, and a reduction layer.

[0043] Among them, the input embedding vector layer (Input embeddings) is used to convert the input data into vector form. For example, Input embeddings can convert words, oranges or documents into vector form; or, Input embeddings can extract features and reduce the dimension of images to obtain vectors; or, Input embeddings can extract the relationship between multiple entities to obtain vectors. The input embedding vector obtained by Input embeddings can first be calculated by the multi-head attention mechanism to obtain the results of each head, and then the results are connected together. The results after the multi-head attention mechanism is processed can be used to calculate the weight scores of each expert through gated matrix multiplication. Then, the activated experts and their weight scores are screened out through the activation function and normalization layer. Finally, the feature vector of each position is nonlinearly transformed through the expert multi-layer perceptron layer to obtain the expert recognition result.

[0044] In the embodiment of the present invention, the expert multi-layer perceptron layer is performed in parallel on multiple devices and multiple cores in the devices, and the calculation results of multiple devices and multiple cores are aggregated through protocols and layers.

[0045] In an embodiment of the present invention, the input tensor of the expert multilayer perceptron layer (expert ffn) can be the output result of the activation function and the normalization layer in the above-mentioned processing. The segmentation of the input tensor of the expert multilayer perceptron layer can be divided into two levels, that is, firstly, the input tensor is segmented according to the computing device data to obtain the preliminary tensor segmentation results corresponding to each computing device; then, the preliminary tensor segmentation results allocated to each computing device are segmented according to the number of cores in the computing device to obtain the final tensor segmentation results corresponding to each core. Through two-layer segmentation, the processing tasks on each computing device and each core can be balanced, so that each computing device and each core can perform calculations in parallel, thereby improving the reasoning speed.

[0046] Step 130: distribute each tensor segmentation result to each core of each computing device, and perform parallel expert multi-layer perceptron layer calculation in each core of each computing device according to the distributed tensor segmentation result to obtain an expert weighted result.

[0047] Among them, the expert multi-layer perceptron layer on each computing device and each core will be adaptively adjusted according to the tensor segmentation results obtained by segmentation, so as to obtain multiple expert weighted results through parallel calculation. It should be noted that there are multiple expert networks that are screened and activated in the expert multi-layer perceptron layer on each computing device and each core. The device core (each computing device and each core) can obtain multiple expert network recognition results through each expert network, and each expert network recognition result can be weighted calculated to obtain an expert weighted result. Weighted summation of the outputs of multiple expert models can effectively integrate the modeling capabilities of each expert and improve the performance and generalization capabilities of the model.

[0048] The weights in weighted calculations come from gated matrix multiplication. The gated network can dynamically adjust the weights of each expert according to the characteristics of the input data to adapt to different data distributions and patterns, thereby improving the flexibility and adaptability of the model. In order to reduce the copy overhead between the device memory and the host memory of the many-core architecture, the present invention implements a weighted sum operator on the many-core computing device to achieve the aggregation of the results on the many-core in the expert dimension.

[0049] Step 140: Reduce and aggregate the expert weighted results in each core of each computing device to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multi-cores.

[0050] Each device core has an expert weighted result. The expert weighted results of multiple device cores can be reduced and aggregated to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous many-cores. The reduction and aggregation can be achieved with the help of a highly optimized communication library on the heterogeneous many-core architecture. Among them, the reduction and aggregation can be based on matrix operations in mathematical algorithms, so that the expert weighted results are aggregated to obtain the result of expert multi-layer perceptron layer calculation based on the original input tensor.

[0051] By splitting the input tensor of the expert multi-layer perceptron layer for parallel calculation, the computing workload is evenly divided among multiple many-core computing devices, giving full play to the parallelism of the heterogeneous many-core architecture.

[0052] The technical solution of this embodiment obtains the number of computing devices in a heterogeneous many-core architecture and the number of cores in each computing device; according to the number of computing devices and the number of cores in each computing device, the input tensor of the expert multi-layer perceptron layer is split to obtain multiple tensor split results; each tensor split result is distributed to each core of each computing device, and in each core of each computing device, parallel expert multi-layer perceptron layer calculation is performed according to the distributed tensor split results to obtain expert weighted results; the expert weighted results in each core of each computing device are reduced and aggregated to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous many-cores, which solves the problem of poor performance of the prior art when using pipeline parallelism for reasoning. By splitting the tensor and placing the calculation on the appropriate computing device core, multiple highly optimized parallel operators can be implemented to complete end-to-end efficient reasoning.

[0053] In an optional implementation of an embodiment of the present invention, before the input tensor of the expert multi-layer perceptron layer is split according to the number of computing devices and the number of cores in each computing device to obtain multiple tensor splitting results, it also includes: according to the number of computing devices and the number of cores in each computing device, the multi-head attention layer is evenly divided, and parallel calculation is performed in each core of each computing device; the parallel attention calculation results of each computing device are aggregated to obtain the overall multi-head attention output result, so as to further obtain the input tensor of the expert multi-layer perceptron layer.

[0054] Among them, each attention head in the multi-head attention layer under the heterogeneous multi-core architecture can be regarded as an independent computing unit, and each attention head is assigned to different device cores for parallel computing. Each device core is responsible for processing the calculation of one or more attention heads. When performing balanced division, you can first divide according to the computing device data, and then divide according to the core data in the computing device to improve the balanced performance of the device. At the same time, the key-value cache (KV Cache) is optimized, and each device core only saves the KV Cache of the divided attention head. Finally, the communication library is used to aggregate the results calculated on each device core to obtain the overall multi-head attention output result.

[0055] On the basis of the above implementation, optionally, after aggregating the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, it also includes: according to the overall multi-head attention output result, repeating the gated matrix multiplication calculation and the activation function and normalization layer calculation in each core of each computing device to obtain the input tensor of the expert multi-layer perceptron layer.

[0056] The gated matrix multiplication (gated GEMM) is used to calculate the weight score of each expert network. Since the number of expert networks in the hybrid expert model is generally small, and the hidden layer dimensions of the model are mostly 4096, 8192, etc., the amount of calculation in this part is not large. If tensor segmentation and parallel calculation are performed, the communication overhead introduced will be greater than the calculation overhead. Therefore, in the embodiment of the present invention, each device core can repeat the calculation of this part and use highly optimized device core code to implement it.

[0057] The activation function and the normalization layer (Softmax-TopK-Normalize) calculation function are used to screen out the activated experts and their weight scores. Since the number of experts activated at the same time is generally smaller, the amount of calculation in this part is very small. In the embodiment of the present invention, there is no need to perform multi-core parallel computing, that is, the calculation can be completed directly in the main core, thereby obtaining the input tensor of the expert multi-layer perceptron layer.

[0058] Figure 2 FIG. 1 is a flowchart of another large-scale hybrid expert language model reasoning method based on heterogeneous multi-core according to the first embodiment of the present invention. Figure 2 As shown in the figure, the input on the device core is calculated in parallel by multiple device cores of the multi-head attention layer after the input embeddings, and then aggregated (all reduce) to generate the overall multi-head attention output result. After that, by repeatedly performing gated matrix multiplication calculations and activation function and normalization layer calculations on each device core, multiple expert network models and their corresponding weights are screened out. Then, the expert multi-layer perceptron layer calculation is performed in parallel on each device core. Among them, the calculation results of multiple expert networks on a single device core can be weighted by the expert network weights. The expert weighted results on each device core can be aggregated by reduction and (all reduce) to obtain the reasoning results of a large-scale hybrid expert language model based on heterogeneous multi-cores. By using tensors for parallel calculations in multi-head attention and expert ffn, the intermediate tensor is divided into multiple small tensors. After the calculation is completed, the communication library is used to perform reduction operations to aggregate the results together, and finally a highly parallel forward reasoning calculation is realized. By splitting the intermediate tensor and adapting multi-head attention and expert ffn, the calculation is set in the most suitable computing device core. By making rational use of these heterogeneous computing resources, different computing tasks can be performed simultaneously on different types of processor cores to improve the speed and efficiency of reasoning. At the same time, the present invention can be easily adapted to be expanded to large-scale heterogeneous multi-core computing clusters to cope with larger model parameter quantities and computing requirements.

[0059] Figure 31 is a rendering of a large-scale hybrid expert language model reasoning method based on heterogeneous multi-core provided in accordance with the first embodiment of the present invention. Figure 3 As shown, the horizontal axis is the inference time, and the vertical axis is the inference of different models at different batch sizes. Among them, tecoinferencelm uses the large-scale hybrid expert language model inference method based on heterogeneous multi-core provided by the embodiment of the present invention for inference, and trt_llm A 800NVLINK, vllm, and huggingface are comparison algorithms. According to Figure 3 As shown, it can be seen that the reasoning speed of the method provided by the embodiment of the present invention has better performance.

[0060] Embodiment 2

[0061] Figure 4 Schematic diagram of a large-scale hybrid expert language model reasoning architecture based on heterogeneous multi-core provided according to the second embodiment of the present invention. Figure 4 As shown, the reasoning architecture includes multiple computing devices, multiple cores in each computing device, and a layer in a large-scale hybrid expert language model is set on each core of each computing device; wherein:

[0062] Each core of each computing device is used to obtain a target tensor segmentation result obtained by segmenting according to the number of computing devices and the number of cores in each computing device, and perform parallel expert multi-layer perceptron layer calculation according to the target tensor segmentation result to obtain an expert weighted result;

[0063] The main core in each computing device is used to reduce and aggregate the expert weighted results in each core of each computing device to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multi-cores.

[0064] Optionally, each core of each computing device is also used to obtain the result of balanced division of the multi-head attention layer according to the number of computing devices and the number of cores in each computing device before obtaining the target tensor segmentation result, and perform multi-head attention layer calculations in parallel; the main core in each computing device is used to aggregate the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, so as to further obtain the input tensor of the expert multi-layer perceptron layer.

[0065] Optionally, each core of each computing device is also used to repeat the gated matrix multiplication calculation and the activation function and normalization layer calculation according to the overall multi-head attention output result to obtain the input tensor of the expert multi-layer perceptron layer.

[0066] Optional, large-scale hybrid expert language model, including: input embedding vector layer, multi-head attention layer, gated matrix multiplication, activation function and normalization layer, expert multi-layer perceptron layer, and reduction layer.

[0067] The technical solution of the embodiment of the present invention can improve the reasoning speed and efficiency by setting the calculation in the most suitable computing device core, making reasonable use of these heterogeneous computing resources, and executing different computing tasks on different types of processor cores at the same time. In addition, based on the repeated gated matrix multiplication calculation and activation function and normalization layer calculation in each core of each computing device, the data dependency between multiple device cores can be reduced, and each computing device core can perform overall parallel calculation without waiting due to data dependency, causing reasoning delay.

[0068] Embodiment 3

[0069] Figure 5 1 is a schematic diagram of a large-scale hybrid expert language model reasoning device based on heterogeneous multi-core according to Embodiment 3 of the present invention. Figure 5 As shown, the device includes: a quantity acquisition module 510, a tensor segmentation module 520, a tensor allocation module 530 and a specification aggregation module 540. Among them:

[0070] A quantity acquisition module 510 is used to acquire the number of computing devices in the heterogeneous many-core architecture and the number of cores in each computing device;

[0071] A tensor splitting module 520 is used to split the input tensor of the expert multi-layer perceptron layer according to the number of computing devices and the number of cores in each computing device to obtain multiple tensor splitting results;

[0072] A tensor allocation module 530 is used to allocate each tensor segmentation result to each core of each computing device, and perform parallel expert multi-layer perceptron layer calculation in each core of each computing device according to the allocated tensor segmentation result to obtain an expert weighted result;

[0073] The reduction and aggregation module 540 is used to reduce and aggregate the expert weighted results in each core of each computing device to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multiple cores.

[0074] Optionally, the device further includes:

[0075] The multi-head attention layer division module is used to divide the input tensor of the expert multi-layer perceptron layer according to the number of computing devices and the number of cores in each computing device, and before obtaining multiple tensor division results, divide the multi-head attention layer evenly according to the number of computing devices and the number of cores in each computing device, and perform parallel calculation in each core of each computing device;

[0076] The attention result aggregation module is used to aggregate the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, so as to further obtain the input tensor of the expert multi-layer perceptron layer.

[0077] Optionally, the device further includes:

[0078] The input tensor determination module is used to aggregate the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, and then repeat the gated matrix multiplication calculation and the activation function and normalization layer calculation in each core of each computing device according to the overall multi-head attention output result to obtain the input tensor of the expert multi-layer perceptron layer.

[0079] Optional, large-scale hybrid expert language model, including: input embedding vector layer, multi-head attention layer, gated matrix multiplication, activation function and normalization layer, expert multi-layer perceptron layer, and reduction layer.

[0080] The large-scale hybrid expert language model reasoning device based on heterogeneous multi-cores provided in an embodiment of the present invention can execute the large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0081] Embodiment 4

[0082] Figure 6 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0083] like Figure 6 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0084] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0085] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as a large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores.

[0086] In some embodiments, the large-scale hybrid expert language model reasoning method based on heterogeneous many cores may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the large-scale hybrid expert language model reasoning method based on heterogeneous many cores described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the large-scale hybrid expert language model reasoning method based on heterogeneous many cores in any other appropriate manner (e.g., by means of firmware).

[0087] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0088] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0089] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0090] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0091] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0092] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0093] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0094] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large-scale hybrid expert language model reasoning method based on heterogeneous many-cores, characterized in that: include: Obtain the number of computing devices in the heterogeneous many-core architecture and the number of cores in each computing device; According to the number of computing devices and the number of cores in each computing device, the input tensor of the expert multi-layer perceptron layer is divided to obtain multiple tensor division results; Distribute each tensor segmentation result to each core of each computing device, and perform parallel expert multi-layer perceptron layer calculation in each core of each computing device according to the distributed tensor segmentation result to obtain an expert weighted result; The expert weighted results in each core of each computing device are reduced and aggregated to obtain the reasoning results of a large-scale hybrid expert language model based on heterogeneous multi-cores.

2. The method according to claim 1, characterized in that Before dividing the input tensor of the expert multi-layer perceptron layer according to the number of computing devices and the number of cores in each computing device to obtain a plurality of tensor division results, the method further includes: According to the number of computing devices and the number of cores in each computing device, the multi-head attention layer is evenly divided and parallelized in each core of each computing device; Aggregate the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, so as to further obtain the input tensor of the expert multi-layer perceptron layer.

3. The method according to claim 2, characterized in that After aggregating the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, it also includes: According to the overall multi-head attention output results, the gated matrix multiplication calculations, activation function and normalization layer calculations are repeated in each core of each computing device to obtain the input tensor of the expert multi-layer perceptron layer.

4. The method according to claim 1, characterized in that: A large-scale hybrid expert language model, including: input embedding vector layer, multi-head attention layer, gated matrix multiplication, activation function and normalization layer, expert multi-layer perceptron layer, and reduction layer.

5. A large-scale hybrid expert language model reasoning architecture based on heterogeneous many-cores, characterized by: The reasoning architecture includes a plurality of computing devices, each computing device includes a plurality of cores, and a layer in a large-scale hybrid expert language model is set on each core of each computing device; wherein: Each core of each computing device is used to obtain a target tensor segmentation result obtained by segmenting according to the number of computing devices and the number of cores in each computing device, and perform parallel expert multi-layer perceptron layer calculation according to the target tensor segmentation result to obtain an expert weighted result; The main core in each computing device is used to reduce and aggregate the expert weighted results in each core of each computing device to obtain a large-scale hybrid expert language model reasoning result based on heterogeneous multi-cores.

6. The reasoning architecture according to claim 5, characterized in that: Each core of each computing device is further used to obtain the result of balanced division of the multi-head attention layer according to the number of computing devices and the number of cores in each computing device before obtaining the target tensor segmentation result, and perform multi-head attention layer calculation in parallel; The main core in each computing device is used to aggregate the parallel attention calculation results of each computing device to obtain the overall multi-head attention output result, so as to further obtain the input tensor of the expert multi-layer perceptron layer.

7. The reasoning architecture according to claim 6, characterized in that: Each core of each computing device is also used to repeatedly perform gated matrix multiplication calculations, activation function and normalization layer calculations based on the overall multi-head attention output results to obtain the input tensor of the expert multi-layer perceptron layer.

8. The reasoning architecture according to claim 5, characterized in that: The large-scale hybrid expert language model includes: an input embedding vector layer, a multi-head attention layer, a gated matrix multiplication, an activation function and normalization layer, an expert multi-layer perceptron layer, and a reduction layer.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the large-scale hybrid expert language model reasoning method based on heterogeneous multi-cores as described in any one of claims 1-4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the large-scale hybrid expert language model reasoning method based on heterogeneous many-cores according to any one of claims 1 to 4 when executed.

Images