Large model reasoning system and tool chain software

Abstract

The invention provides a large model reasoning system and tool chain software. The system at least comprises a first processor, a second processor and at least one processing chip, the second processor is responsible for calculation of a first part of reasoning of the large model, and the processing chip is responsible for storing weight parameters and calculation of a second part of reasoning of the large model, so that on one hand, repeated carrying of the weight parameters can be avoided, and on the other hand, all calculations of the large model can be prevented from being gathered in the same device; therefore, power consumption and calculation time delay are reduced.

Description

Technical Field

[0001] This invention relates to the field of semiconductor technology, and more specifically to a large-scale model inference system and toolchain software. Background Technology

[0002] Large model inference is a rapidly growing AI computing need, and it is currently performed using graphics processing unit (GPU) servers or central processing unit (CPU) servers.

[0003] However, whether large model inference is performed using GPU servers or CPU servers, both GPU servers and CPU servers need to repeatedly move the weight parameters of the large model from memory to perform large model inference, which will result in huge energy consumption and high computational latency. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a large model inference system and toolchain software to solve the problems of high energy consumption and high computational latency in the current method of performing large model inference by GPU servers or CPU servers.

[0005] To achieve the above objectives, the embodiments of the present invention provide the following technical solutions:

[0006] The first aspect of this invention discloses a large model inference system, the system comprising at least: a first processor, a second processor, and at least one processing chip;

[0007] The first processor is configured to distribute a first hidden layer vector calculated by the second processor to the processing chip, and to send a second hidden layer vector calculated by the processing chip to the second processor;

[0008] The processing chip is used to perform second part inference of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor in the first part of inference of the (i-1)-th layer of the large model.

[0009] The second processor is configured to perform a first part of inference of the (i+1)th layer of the large model based on the second hidden layer vector of the i-th layer to obtain the first hidden layer vector of the (i+1)th layer.

[0010] Preferably, the first part of the reasoning is an attention mechanism computation, and the second part of the reasoning is a forward propagation computation.

[0011] Preferably, the processing chip is integrated from at least memory chip particles and logic chip particles;

[0012] The memory chip particles are used to store the weight parameters of the large model;

[0013] The logic chip particle is used to perform a second part of the reasoning of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor in the first part of the reasoning of the (i-1)-th layer of the large model.

[0014] Preferably, the process by which the logic chip particle performs the second part of the inference of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer includes:

[0015] The weight parameters of the large model are read from the memory chip particles;

[0016] Based on the weight parameters and the first hidden layer vector obtained by the second processor in the first part of the inference of the (i-1)th layer of the large model, the second part of the inference of the i-th layer of the large model is performed to obtain the second hidden layer vector of the i-th layer.

[0017] Preferably, the memory chip is a 3D-NAND chip.

[0018] Preferably, the first processor is specifically used to: distribute the first hidden layer vector calculated by the second processor to the processing chip through a specified transmission mode, wherein the specified transmission mode is a broadcast mode, a point-to-point transmission mode, or a self-organizing network relay transmission mode.

[0019] Preferably, the logic chip particle is further configured to: provide voltage to the memory chip particle.

[0020] Preferably, the processing chip further includes a voltage chip;

[0021] The voltage chip is used to provide voltage to the memory chip particles and the logic chip particles.

[0022] Preferably, the first processor is a CPU, NPU, or DPU, and the second processor is a GPU, NPU, or DPU.

[0023] A second aspect of this invention discloses a toolchain software, which includes at least a software layer, a tool layer, and a hardware layer;

[0024] The software layer is provided with neural network software development tools, the tool layer is provided with deployment tools, and the hardware layer is provided with a large model inference system disclosed in the first aspect of the present invention, which is compatible with the deployment tools.

[0025] Based on the above embodiments of the present invention, a large model inference system and toolchain software are provided. The system includes at least: a first processor, a second processor, and at least one processing chip. The first processor distributes a first hidden layer vector calculated by the second processor to the processing chip, and sends a second hidden layer vector calculated by the processing chip to the second processor. The processing chip, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor during the first part of the inference of the (i-1)th layer of the large model, performs the second part of the inference of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer. The second processor, based on the second hidden layer vector of the i-th layer, performs the first part of the inference of the i+1th layer of the large model to obtain the first hidden layer vector of the i+1th layer. In this scheme, the second processor is responsible for calculating the first part of the inference of the large model, and the processing chip is responsible for storing the weight parameters and calculating the second part of the inference of the large model. This avoids repeated handling of weight parameters and also prevents all calculations of the large model from being concentrated in the same device, thereby reducing power consumption and computational latency. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0027] Figure 1 A structural block diagram of a large model inference system provided in an embodiment of the present invention;

[0028] Figure 2 Another structural block diagram of a large model inference system provided in an embodiment of the present invention;

[0029] Figure 3 An example diagram of the architecture of a processing chip provided in an embodiment of the present invention;

[0030] Figure 4 Another structural block diagram of a large model inference system provided in an embodiment of the present invention;

[0031] Figure 5 This is another architectural example diagram of the processing chip provided in an embodiment of the present invention;

[0032] Figure 6 An example diagram illustrating the distribution of the first hidden layer vector provided in an embodiment of the present invention;

[0033] Figure 7This is an example diagram of the overall architecture of a large model inference system provided in an embodiment of the present invention;

[0034] Figure 8 This is a structural block diagram of a toolchain software provided in an embodiment of the present invention. Detailed Implementation

[0035] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0036] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0037] To better understand the following content, the following abbreviations will be explained: Graphics Processing Unit (GPU), Embedded Neural Processing Unit (NPU), Central Processing Unit (CPU), Data Processing Unit (DPU), and Multiply Accumulate (MAC).

[0038] Currently, large model inference is usually performed using GPU servers or CPU servers. However, regardless of whether GPU servers or CPU servers are used for large model inference, both GPU servers and CPU servers need to repeatedly move the weight parameters of the large model from memory for large model inference. Specifically, GPU servers repeatedly move the weight parameters from video memory, while CPU servers repeatedly move the weight parameters from DRAM. This results in huge energy consumption and high computational latency.

[0039] In addition, all computations for large model inference are concentrated on CPU servers or GPU servers, resulting in concentrated computational burden, power consumption, and heat dissipation issues.

[0040] To address this, this invention proposes a large-model inference system and toolchain software. The system includes at least a first processor, a second processor, and at least one processing chip. The second processor is responsible for calculating the first part of the large model's inference, while the processing chip is responsible for storing weight parameters and calculating the second part of the large model's inference. This avoids the repeated transfer of weight parameters and prevents all calculations of the large model from being concentrated in the same device, thereby reducing power consumption and computational latency.

[0041] In practical applications, one application scenario for this solution is as follows: for large-capacity, low-cost memory chips such as 3D-NAND chips, these memory chips can replace DRAM and video memory to store weight parameters. The memory chips and logic chips with logic operation functions can be integrated together to form a processing chip. The processing chip and the second processor are responsible for part of the inference tasks of the large model, which can reduce power consumption and computation latency, as well as reduce memory usage and cost.

[0042] First, it should be noted that the reasoning process of the large model is divided into two parts: an alternating stack of first-part reasoning and second-part reasoning.

[0043] In some specific embodiments, the first part of the inference is attention mechanism computation, and the second part of the inference is feed-forward computation (also known as expert model). The inference process of the large model is divided into alternating stacked "Attention→Feed-forward→Attention→Feed-forward…".

[0044] See Figure 1 The diagram shows a structural block diagram of a large model inference system provided by an embodiment of the present invention. The large model inference system includes: a first processor 101, a second processor 102, and at least one processing chip 103.

[0045] Specifically, the first processor 101 is connected to each of the processing chips 103, and the first processor 101 is connected to the second processor 102 via a bus (e.g., CXL data bus).

[0046] The first processor 101 is configured to distribute the first hidden layer vector calculated by the second processor 102 to the processing chip 103, and to send the second hidden layer vector calculated by the processing chip 103 to the second processor 102.

[0047] The processing chip 103 is used to perform the second part of the reasoning of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor 102 in the first part of the reasoning of the (i-1)-th layer of the large model.

[0048] The second processor 102 is used to perform a first part of the reasoning of the (i+1)th layer of the large model based on the second hidden layer vector of the i-th layer to obtain the first hidden layer vector of the (i+1)th layer.

[0049] Specifically, the processing chip 103 stores the weight parameters of the large model and deploys the "second part of the large model inference" function (i.e., deploys the feed-forward calculation function), while the second processor 102 deploys the "first part of the large model inference" function (i.e., deploys the attention calculation function).

[0050] The second processor 102 performs the first part of the large model inference to obtain the first hidden layer vector, and then the second processor 102 sends the first hidden layer vector to the first processor 101; the first processor 101 distributes the received first hidden layer vector to the processing chip 103; the processing chip 103 performs the second part of the large model inference according to the received first hidden layer vector to obtain the second hidden layer vector; the first processor 101 continues to send the second hidden layer vector back to the second processor 102 to perform the first part of the inference for the next layer.

[0051] In some specific embodiments, the first processor 101 is a CPU, NPU, or DPU, and the second processor 102 is a GPU, NPU, or DPU.

[0052] In this embodiment of the invention, the second processor is responsible for the calculation of the first part of the inference of the large model, and the processing chip is responsible for storing the weight parameters and the calculation of the second part of the inference of the large model. On the one hand, it can avoid the repeated transfer of weight parameters, and on the other hand, it can avoid all the calculations of the large model being gathered in the same device, thereby reducing power consumption and computation latency.

[0053] In some specific embodiments, see Figure 2 , Figure 2 This diagram illustrates another structural block diagram of a large model inference system provided by an embodiment of the present invention, wherein the processing chip 103 is integrated from at least a memory chip 1031 and a logic chip 1032.

[0054] The 1031 memory chip is used to store the weight parameters of large models.

[0055] The logic chip 1032 is used to perform the second part of the reasoning of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor in the first part of the reasoning of the (i-1)-th layer of the large model.

[0056] In other words, the storage chip 1031 in the processing chip 103 provides storage functionality to store the weight parameters of the large model, and the logic chip 1032 in the processing chip 103 provides at least logical computation functionality to implement the "second part inference" operation. That is, the weight parameters of the large model are stored in the processing chip 103, and the "second part inference based on the weight parameters" is also deployed in the processing chip 103. The processing chip 103 has both storage and logical computation functions.

[0057] In a practical application scenario, the large model inference system of this solution has multiple processing chips 103, and each processing chip 103 is deployed with the second part of the large model inference (feed-forward computation, also known as expert model), that is, multiple expert models are deployed on different processing chips 103.

[0058] In the above application scenario, the process of large model inference is as follows: the processing chip 103 performs feed-forward calculation according to the weight parameters to obtain the second hidden layer vector; through the interconnection between the first processor 101, the second processor 102 and the processing chip 103, the second hidden layer vector is sent back to the second processor 102 for the first part of the inference (Attention calculation) of the next layer; the first part of the inference is executed in the second processor 102 to obtain the first hidden layer vector; through the interconnection between the first processor 101, the second processor 102 and the processing chip 103, the first hidden layer vector is sent to one or more processing chips 103 for the feed-forward calculation of the next layer.

[0059] In some specific embodiments, the process of the logic chip 1032 performing the second part of inference is as follows: read the weight parameters of the large model from the storage chip 1031; and perform the second part of inference of the i-th layer of the large model according to the weight parameters and the "first hidden layer vector obtained by the second processor 102 in performing the first part of inference of the i-1th layer of the large model" to obtain the second hidden layer vector of the i-th layer.

[0060] In some specific embodiments, the memory chip 1031 may be a 3D-NAND chip or other chip with a memory array. The specific type of memory chip 1031 is not limited here.

[0061] It should be noted that the method of integrating the memory chip 1031 and the logic chip 1032 to obtain the processing chip 103 is as follows: the memory chip 1031 with a memory array (e.g., a 3D-NAND memory array) and the logic chip 1032 with logic control function, data read and write function and calculation function are stacked together by 3D packaging to form the processing chip 103.

[0062] Let's take the 1031 memory chip as an example of a 3D-NAND chip. Figure 3 The illustrated architecture diagram of the processing chip provides an example of the memory chip 1031 and logic chip 1032 that make up the processing chip 103.

[0063] like Figure 3 As shown, in each processing chip 103, the 3D-NAND chip (storage chip 1031) is used to store the weight parameters of the large model; the logic chip 1032 includes the following components: a control circuit for writing and reading data from the 3D-NAND chip, a communication interface circuit (IO circuit) for receiving the first hidden layer vector to be calculated and outputting the second hidden layer vector, an adder and a multiplier (e.g., a MAC circuit) for performing parallel matrix multiplication, and a matching cache circuit.

[0064] The 3D-NAND chip and logic chip 1032 are integrated together through stacking, TSV and other methods to form the processing chip 103 of this solution.

[0065] It should be noted that, compared to ordinary chip chips, the 3D-NAND chip chips used in this solution have a higher number of bit lines and a shorter data read time.

[0066] In some specific embodiments, the logic chip 1032 is also used to: provide voltage (provide high voltage) to the memory chip 1031.

[0067] For example Figure 3 As can be seen from the example diagram of the processing chip architecture, the logic chip 1032 itself has circuitry that provides high voltage to the memory chip 1031 (3D-NAND chip).

[0068] Understandably, in some scenarios, the logic chip 1032 itself does not have the circuitry to provide high voltage. In such cases, high voltage can be provided by encapsulating a third voltage chip (i.e., a high-voltage chip).

[0069] In some specific embodiments, see Figure 4 , Figure 4This diagram illustrates another structural block diagram of a large model inference system provided in an embodiment of the present invention. The processing chip 103 also includes a voltage chip 1033.

[0070] Voltage chip 1033 is used to provide voltage to memory chip 1031 and logic chip 1032.

[0071] For example Figure 5 As shown in another example diagram of the processing chip architecture, a third voltage chip 1033 is used to provide high voltage to the 3D-NAND chip (memory chip 1031) and logic chip 1032.

[0072] As can be seen from the above embodiments, the first part of the large model inference (Attention calculation) is performed on the second processor 102. The one or more sets of first hidden layer vectors generated by the first part of the inference need to be distributed to the processing chip 103 for the second part of the large model inference (Feed-forward calculation). This distribution process needs to be completed by a routing component.

[0073] In some specific embodiments, the first processor 101 is specifically used to: distribute the first hidden layer vector calculated by the second processor 102 to the processing chip 103 by specifying a transmission mode, wherein the specified transmission mode is a broadcast mode, a point-to-point transmission mode, or a self-organizing network relay transmission mode, and the self-organizing network relay transmission mode is the self-organizing network relay transmission mode of the processing chip 103.

[0074] In other words, the first processor 101 acts as a routing component to complete the distribution of the first hidden layer vector. The first processor 101 can be a CPU, NPU, or DPU.

[0075] For example Figure 6 As shown in the example diagram illustrating the distribution of the first hidden layer vector, the first processor 101 sends the first hidden layer vector to one or more processing chips 103 using a broadcast mode, a point-to-point transmission mode, or a self-organizing network relay transmission mode.

[0076] It should be noted that the physical path by which the first processor 101 distributes the first hidden layer vector to the processing chip 103 can be: DDR interface, PCIe interface, high-speed network port, wireless network or other high-speed parallel interface.

[0077] In addition, the first processor 101 also has the function of one-to-one and one-to-many access to the processing chip 103, and can read the second hidden layer vector calculated by the processing chip 103.

[0078] It should be noted that in practical applications, the first processor 101 mentioned in this solution also has exception handling function, token scheduling function, etc. The functions of the first processor 101 will not be illustrated one by one here.

[0079] Applying the above embodiments, one application scenario for this solution is as follows: See Figure 7 The diagram illustrates an example of the overall architecture of a large model inference system. This system includes: a first processor 101, a second processor 102, multiple processing chips 103, a video memory 104, a memory 105, and a data bus 106. The first processor 101 is a CPU (connected to SRAM), the second processor 102 is a GPU, the memory 105 is DRAM used to store intermediate data, the processing chips 103 are integrated from "3D-NAND chip particles storing weight parameters" and "logic chip particles 1032 deployed with feed-forward computation," and the data bus 106 is a CXL data bus.

[0080] It should be noted that SRAM stands for Static Random Access Memory, while DRAM stands for Dynamic Random Access Memory.

[0081] As can be seen from the above embodiments, this solution deploys "Feed-forward computation" on the processing chip 103 and "Attention computation" on the second processor 102. Therefore, the large model inference system proposed in this solution is an AF-separated system structure, where A represents Attention computation and F represents Feed-forward computation.

[0082] Therefore, this solution has the following beneficial effects: the weight parameters of the large model are stored in the processing chip 103, and the feed-forward calculation is deployed on the processing chip 103. When performing the feed-forward calculation, there is no need to move the weight parameters back and forth, saving the power consumption and time of data transportation; the feed-forward calculation is performed on the processing chip 103, freeing up the computing power of the first processor 101, so that the first processor 101 can focus on the scheduling of computing tasks, and the computing speed of the feed-forward calculation is no longer limited by the computing power of the first processor 101.

[0083] Preferred, see Figure 8 The present invention also provides a structural block diagram of toolchain software, which includes at least a software layer 801, a tool layer 802 and a hardware layer 803.

[0084] Specifically, the software layer 801 is provided with neural network software development tools, the tool layer 802 is provided with deployment tools, and the hardware layer 802 is provided with the large model inference system provided in the above-described embodiments of the present invention, which is compatible with the deployment tool 8021.

[0085] Specifically, the tool layer 802 is equipped with multiple deployment tools, which are used to deploy various systems. For the large model inference system (AF-separated system architecture) provided by the embodiments of the present invention, which is set in the hardware layer 802, the large model inference system provided by the embodiments of the present invention can be compatible with the various deployment tools in the tool layer 802.

[0086] The specific details of the large-scale model inference system have been described in detail in the above embodiments and will not be repeated here.

[0087] In summary, the embodiments of the present invention provide a large model inference system and toolchain software. The second processor is responsible for the calculation of the first part of the large model inference, and the processing chip is responsible for storing weight parameters and calculating the second part of the large model inference. On the one hand, it can avoid the repeated transfer of weight parameters, and on the other hand, it can avoid all the calculations of the large model being gathered in the same device, thereby reducing power consumption and computation latency.

[0088] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for system or system embodiments, since they are basically similar to method embodiments, the description is relatively simple, and relevant parts can be referred to the descriptions in the method embodiments. The systems and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0089] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0090] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A large-scale model reasoning system, characterized in that, The system includes at least: a first processor, a second processor, and at least one processing chip; The first processor is configured to distribute a first hidden layer vector calculated by the second processor to the processing chip, and to send a second hidden layer vector calculated by the processing chip to the second processor; The processing chip is used to perform second part inference of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor in the first part of inference of the (i-1)-th layer of the large model. The second processor is configured to perform a first part of inference of the (i+1)th layer of the large model based on the second hidden layer vector of the i-th layer to obtain the first hidden layer vector of the (i+1)th layer.

2. The system according to claim 1, characterized in that, The first part of the reasoning is an attention mechanism computation, and the second part of the reasoning is a forward propagation computation.

3. The system according to claim 1, characterized in that, The processing chip is at least integrated from memory chip particles and logic chip particles; The memory chip particles are used to store the weight parameters of the large model; The logic chip particle is used to perform a second part of the reasoning of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer, based on the weight parameters of the large model and the first hidden layer vector obtained by the second processor in the first part of the reasoning of the (i-1)-th layer of the large model.

4. The system according to claim 3, characterized in that, The process by which the logic chip particle performs the second part of the inference of the i-th layer of the large model to obtain the second hidden layer vector of the i-th layer includes: The weight parameters of the large model are read from the memory chip particles; Based on the weight parameters and the first hidden layer vector obtained by the second processor in the first part of the inference of the (i-1)th layer of the large model, the second part of the inference of the i-th layer of the large model is performed to obtain the second hidden layer vector of the i-th layer.

5. The system according to claim 3, characterized in that, The memory chip is a 3D-NAND chip.

6. The system according to claim 1, characterized in that, The first processor is specifically used to: distribute the first hidden layer vector calculated by the second processor to the processing chip through a specified transmission mode, wherein the specified transmission mode is a broadcast mode, a point-to-point transmission mode, or a self-organizing network relay transmission mode.

7. The system according to claim 3, characterized in that, The logic chip particle is also used to: provide voltage to the memory chip particle.

8. The system according to claim 3, characterized in that, The processing chip also includes: a voltage chip; The voltage chip is used to provide voltage to the memory chip particles and the logic chip particles.

9. The system according to claim 1, characterized in that, The first processor is a CPU, NPU, or DPU, and the second processor is a GPU, NPU, or DPU.

10. A toolchain software, characterized in that, The toolchain software includes at least a software layer, a tool layer, and a hardware layer; The software layer is provided with neural network software development tools, the tool layer is provided with deployment tools, and the hardware layer is provided with a large model inference system as described in any one of claims 1-9 above, which is compatible with the deployment tools.

Images