Method for processing computing task of ai model, and inference server

Abstract

The present application relates to the technical field of computers, and discloses a method for processing a computing task of an AI model, and an inference server. The device comprises at least one chipset, the chipset is obtained by bonding a computing chip and a memory chip, and the computing chip comprises a first operator. The computing chip is used for processing a first portion of a computing task of an AI model by means of the first operator. The memory chip is used for storing data related to the first portion. The present application does not require any modifications within either a computing chip or a memory chip, the computing chip and the memory chip are produced according to appropriate processes, and then are bonded together to form a chipset, such that both the area and computing power of the computing chip, as well as the area, capacity and bandwidth of the memory chip are taken into account, allowing the chipset to have good characteristics in terms of computing power, capacity and bandwidth and to flexibly and efficiently process computing tasks. The computing tasks that the device can process include, but not limited to, inference tasks of AI models.

Description

AI model computational task processing device and inference server

[0001] This application claims priority to Chinese Patent Application No. 202410918498.3, filed on July 9, 2024, entitled "Computational Task Processing Device and Inference Server for AI Model", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of computer technology, and in particular to computing task processing devices and inference servers for AI models. Background Technology

[0003] In the field of computer technology, there is a need to process computational tasks related to artificial intelligence (AI). These AI model computational tasks include, but are not limited to, training and inference tasks. How to flexibly handle the computational tasks of AI models has become a noteworthy issue. Summary of the Invention

[0004] This application provides a computational task processing device and inference server for AI models, enabling flexible processing of computational tasks. The technical solution provided in this application includes the following aspects.

[0005] In a first aspect, this application provides a computational task processing apparatus for an AI model. The apparatus includes at least one chipset, which is formed by bonding a computing chip and a storage chip. The computing chip includes a first operator. The computing chip is used to process a first part of the computational task of the AI ​​model through the first operator. The storage chip is used to store data related to the first part.

[0006] In this application, the computing chip provides computing functionality through a first operator to process the first part of the computational task of the AI ​​model, and the storage chip provides storage functionality to store data related to the first part, including data involved in the processing of the first part, such as the data to be computed in the first part, or the results obtained from computing the data to be computed. Since the chipset is obtained by bonding the computing chip and the storage chip, no internal modifications are required to the computing chip, thus avoiding an increase in the area of ​​the computing chip due to internal modifications, and eliminating the need to sacrifice computing power to maintain the same area. Similarly, no internal modifications are required to the storage chip, thus avoiding an increase in the area of ​​the storage chip, and eliminating the need to sacrifice storage capacity and internal bandwidth to maintain the same area. Furthermore, this application allows for bonding the computing chip and the memory chip after they have been manufactured separately. This enables the computing chip and the memory chip to each employ appropriate processes, resulting in the computing chip possessing strong computing power and the memory chip possessing large capacity and internal bandwidth. This ensures that the chipset obtained by bonding the computing chip and the memory chip meets the actual requirements in terms of computing power, capacity, and internal bandwidth, thereby ensuring that the chipset can flexibly and efficiently complete the processing of AI model computing tasks.

[0007] In one possible implementation, the device further includes a first interface and a control chip. The control chip is used to receive a first portion sent by the processing chip through the first interface. The control chip is also used to acquire the processing result of the first portion and send the processing result of the first portion to the processing chip through the first interface.

[0008] In this implementation, after receiving the first part sent by the processing chip through the first interface, the control chip can send the first part to the chipset, obtain the result of the chipset processing the first part, obtain the processing result of the first part based on the obtained result, and return the processing result of the first part to the processing chip. This implementation enables interaction between the control chip and the processing chip. The processing chip can send computing tasks to the control chip according to actual needs, so as to flexibly handle the computing tasks of the AI ​​model and ensure the processing efficiency of the computing tasks.

[0009] In one possible implementation, the processing chip is also used to identify the first operator, obtain the identification result, and send the first part to the control chip based on the identification result.

[0010] In other words, the first part received by the control chip is the recognition result obtained by the processing chip based on the recognition of the first operator. By recognizing the first operator, the processing capability of the first operator can be determined, thus ensuring that the first part sent to the control chip is the computational task supported by the processing capability of the first operator. This guarantees that the computing chip can perform the processing of the first part through the first operator, avoiding processing failures.

[0011] In one possible implementation, the processing chip includes a second operator. The processing chip is also used to process a second part of the computational task of the AI ​​model via the second operator.

[0012] In other words, the computational task of the AI ​​model consists of a first part and a second part, with the second part processed by the processing chip through a second operator. In this implementation, the chipset (or chipset and control chip) completes one part of the computational task, while the processing chip completes the other part, working together to complete the computational task. This approach is more flexible and helps improve the processing efficiency of the computational task.

[0013] In one possible implementation, the computational power requirement of the first operator is lower than that of the second operator.

[0014] Due to manufacturing processes and other reasons, the computing power provided by the chipset may be less than that provided by the processing chip. If the computing power requirement of the first operator is high, the chipset will need to consume a longer time to complete the processing of the first part, reducing the processing efficiency of the first part. Therefore, this application makes the computing power requirement of the first operator lower than that of the second operator to ensure the processing efficiency of the first part, thereby ensuring the processing efficiency of the AI ​​model's computing tasks.

[0015] In one possible implementation, the sum of the data volume corresponding to the first part and the data volume of the second processing result is less than the data volume threshold.

[0016] The sum of the data amounts represents the total amount of data exchanged between the control chip and the processing chip based on the first interface. If the total data amount exceeds a data amount threshold, it indicates that the control chip and the processing chip need to exchange a large amount of data, and the data exchange process may consume a long time. Therefore, this application ensures that the sum of the aforementioned data amounts is less than the data amount threshold, so that the control chip and the processing chip only need to exchange less data, and the data exchange process consumes more time, which is beneficial to ensuring the processing efficiency of the AI ​​model's computational tasks.

[0017] In one possible implementation, the storage capacity requirement of the first operator is higher than that of the second operator.

[0018] The storage capacity provided by the chipset (such as the capacity of the memory chip) may be greater than the storage capacity provided by the processing chip. If the storage capacity requirement of the second operator is high, the processing chip will need to spend a longer time to complete the processing of the second part, reducing the processing efficiency of the second part. Therefore, this application makes the storage capacity requirement of the second operator lower than that of the first operator to ensure the processing efficiency of the second part, thereby ensuring the processing efficiency of the AI ​​model's computational tasks.

[0019] In one possible implementation, the bandwidth requirement of the first operator is higher than that of the second operator.

[0020] The bandwidth provided by the chipset (such as the capacity of the memory chip) may be greater than that provided by the processing chip. If the bandwidth requirement of the second operator is high, the processing chip will need to spend a longer time to complete the processing of the second part, reducing the processing efficiency of the second part. Therefore, this application makes the bandwidth requirement of the second operator lower than that of the first operator to ensure the processing efficiency of the second part, thereby ensuring the processing efficiency of the AI ​​model's computational tasks.

[0021] In one possible implementation, the first part includes a first sub-part and a second sub-part, and the control chip includes a third operator. A computing chip is used to process the first sub-part via the first operator. The control chip is also used to process the second sub-part via the third operator. The control chip is used to obtain the processing result of the first part based on the processing results of the first and second sub-parts respectively.

[0022] In this implementation, the chipset and the control chip each complete a part of the first part. That is, the chipset and the control chip work together to complete the processing of the first part, which helps to improve the processing efficiency of the first part and thus improves the processing efficiency of the AI ​​model's computing tasks.

[0023] In one possible implementation, the computation task is an inference task, and the first part consists of a key-value caching task and a numerical computation task. Optionally, the second part differs from the numerical computation task and the key-value caching task.

[0024] For example, in the case where the chipset processes the first sub-part and the control chip processes the second sub-part, the first sub-part is a key-value caching task and the second sub-part is a numerical calculation task. The second part is different from both the numerical calculation task and the key-value caching task.

[0025] In another possible implementation, the computation task is an inference task, and the first part is a key-value caching task. Optionally, the second part is different from the key-value caching task.

[0026] The inference task of the AI ​​model (including but not limited to the basic model, also known as the large model) may include a variety of computational tasks. This application reasonably divides the various computational tasks into a first part and a second part, so that the chipset (or chipset and control chip) and the processing chip can complete the first part and the second part with high efficiency, respectively, to ensure the processing efficiency of the computational tasks.

[0027] In one possible implementation, there are multiple chipsets, with at least two of them stacked on the baseboard.

[0028] With the baseboard area remaining unchanged, stacking at least two chipsets on the baseboard allows for the placement of more chipsets on the baseboard, providing stronger computing power, greater capacity, and internal bandwidth, thus ensuring the processing efficiency of computing tasks.

[0029] In one possible implementation, the bonding method for the computing chip and the memory chip is a hybrid bonding method.

[0030] This bonding method is simple, easy to implement, and widely applicable.

[0031] In one possible implementation, the manufacturing process of the computing chip differs from that of the memory chip.

[0032] The manufacturing processes of computing chips and memory chips do not interfere with each other, enabling computing chips to have strong computing power and memory chips to have large capacity and internal bandwidth. The chipset obtained by bonding computing chips and memory chips can flexibly and efficiently complete computing tasks.

[0033] Secondly, an inference server is provided, comprising a computing task processing device for at least one AI model. The device includes at least one chipset, which is obtained by bonding a computing chip and a storage chip. The computing chip includes a first operator. The computing chip is used to process a first part of the computing task of the AI ​​model through the first operator. The storage chip is used to store data related to the first part.

[0034] In one possible implementation, the device further includes a first interface and a control chip. The control chip is used to receive a first portion sent by the processing chip through the first interface. The control chip is also used to acquire the processing result of the first portion and send the processing result of the first portion to the processing chip through the first interface.

[0035] In one possible implementation, the processing chip is also used to identify the first operator, obtain the identification result, and send the first part to the control chip based on the identification result.

[0036] In one possible implementation, the processing chip includes a second operator. The processing chip is also used to process a second part of the computational task of the AI ​​model via the second operator.

[0037] In one possible implementation, the storage capacity requirement of the first operator is higher than that of the second operator.

[0038] In one possible implementation, the bandwidth requirement of the first operator is higher than that of the second operator.

[0039] In one possible implementation, the computational power requirement of the first operator is lower than that of the second operator.

[0040] In one possible implementation, the sum of the data volume of the first part and the processing result of the first part is less than the data volume threshold.

[0041] In one possible implementation, the first part includes a first sub-part and a second sub-part, and the control chip includes a third operator. A computing chip is used to process the first sub-part via the first operator; the control chip is also used to process the second sub-part via the third operator; and the control chip is used to obtain the processing result of the first part based on the processing results of the first and second sub-parts respectively.

[0042] In one possible implementation, the computation task is a reasoning task. The first part consists of a key-value caching task and a numerical computation task, or the first part consists of a key-value caching task.

[0043] In one possible implementation, the computation task is an inference task, the first sub-part is a key-value caching task, and the second sub-part is a numerical computation task.

[0044] In one possible implementation, the computing chip further includes a second interface, and the storage chip includes a third interface, with the computing chip and the storage chip bonded to each other via the second interface and the third interface.

[0045] Thirdly, a method for processing computational tasks of an AI model is provided. This method can be applied to a device for processing computational tasks of an AI model. The device includes at least one chipset, which is obtained by bonding a computing chip and a storage chip. The computing chip includes a first operator. In this method, the computing chip processes a first part of the computational task of the AI ​​model through the first operator. Data related to the first part is stored through the storage chip.

[0046] In one possible implementation, the apparatus further includes a first interface and a control chip. The method further includes: receiving a first portion sent by a processing chip via the first interface through the control chip; obtaining a processing result of the first portion via the control chip; and sending the processing result of the first portion to the processing chip via the first interface.

[0047] In one possible implementation, the method further includes: identifying the first operator through a processing chip, obtaining an identification result, and sending a first part to the control chip based on the identification result. In other words, the first part received by the control chip is the identification result obtained by the processing chip based on the identification of the first operator.

[0048] In one possible implementation, the processing chip includes a second operator. The method further includes processing a second portion of the computational task of the AI ​​model via the second operator using the processing chip. That is, the computational task of the AI ​​model includes a first part and a second part, with the second part being processed by the processing chip via the second operator.

[0049] In one possible implementation, the storage capacity requirement of the first operator is higher than that of the second operator.

[0050] In one possible implementation, the bandwidth requirement of the first operator is higher than that of the second operator.

[0051] In one possible implementation, the computational power requirement of the first operator is lower than that of the second operator.

[0052] In one possible implementation, the sum of the data volume of the first part and the processing result of the first part is less than the data volume threshold.

[0053] In one possible implementation, the first part includes a first sub-part and a second sub-part, and the control chip includes a third operator. Processing the first part of the computational task of the AI ​​model via the first operator using the computing chip includes: processing the first sub-part via the first operator using the computing chip. The method further includes: processing the second sub-part via the third operator using the control chip. Obtaining the processing result of the first part via the control chip includes: obtaining the processing result of the first part via the control chip based on the respective processing results of the first and second sub-parts.

[0054] In one possible implementation, the computation task is a reasoning task. The first part consists of a key-value caching task and a numerical computation task, or the first part consists of a key-value caching task.

[0055] In one possible implementation, the computation task is an inference task, the first sub-part is a key-value caching task, and the second sub-part is a numerical computation task.

[0056] In one possible implementation, the computing chip further includes a second interface, and the storage chip includes a third interface, with the computing chip and the storage chip bonded to each other via the second interface and the third interface.

[0057] It should be understood that the technical effects achieved by the technical solutions provided by the second and third aspects of this application and their corresponding possible implementations can be found in the above description of the technical effects achieved by the technical solutions provided by the first aspect and their corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0058] Figure 1 is a schematic diagram of the structure of an AI model computing task processing device provided in an embodiment of this application;

[0059] Figure 2 is a schematic diagram of a hybrid bonding method provided in an embodiment of this application;

[0060] Figure 3 is a schematic diagram of the structure of a computing task processing device for another AI model provided in an embodiment of this application;

[0061] Figure 4 is a schematic diagram of the structure of another AI model computing task processing device provided in the embodiment of this application;

[0062] Figure 5 is a schematic diagram of a structure in which different chipsets are arranged in parallel according to an embodiment of this application;

[0063] Figure 6 is a schematic diagram of a stacked arrangement of different chipsets provided in an embodiment of this application;

[0064] Figure 7 is a schematic diagram of the connection between a processing chip and a PIM disk provided in an embodiment of this application;

[0065] Figure 8 is a schematic diagram of the structure of an inference server provided in an embodiment of this application;

[0066] Figure 9 is a schematic diagram of a time ratio provided in an embodiment of this application;

[0067] Figure 10 is a schematic diagram of a GEMM and GEMV provided in an embodiment of this application;

[0068] Figure 11 is a schematic diagram showing the ratio of operation quantity and duration provided in an embodiment of this application;

[0069] Figure 12 is a flowchart illustrating the computational task processing of an AI model according to an embodiment of this application;

[0070] Figure 13 is a schematic diagram of the structure of a chip in a related art provided in an embodiment of this application;

[0071] Figure 14 is a flowchart of a computational task processing method for an AI model provided in an embodiment of this application;

[0072] Figure 15 is a flowchart of another AI model computation task processing method provided in the embodiments of this application. Detailed Implementation

[0073] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0074] With the widespread application of AI technology, the hardware used to support AI models has gradually received significant attention. Both the training and inference processes of AI models involve substantial computational tasks, requiring the hardware to handle at least a portion of these tasks. For example, the computational tasks involved in training are called training tasks, and the computational tasks involved in inference are called inference tasks. The hardware supporting AI models must be capable of handling at least one of these tasks.

[0075] This application provides hardware for supporting AI models, which is a device for processing computational tasks of AI models. As shown in FIG1, the device includes at least one chipset, which is obtained by bonding a computing chip and a storage chip. The computing chip includes (or integrates) a first operator. The computing chip is used to process a first part of the computational task of the AI ​​model through the first operator. The storage chip is used to store data related to the first part.

[0076] For example, the number of first operators is at least one, but the embodiments of this application do not limit the number of first operators. The first operators can be used to process computational tasks of AI models, such as processing at least one of the training or inference tasks of AI models.

[0077] Optionally, the first part of the computational task of the AI ​​model can refer to the entire computational task of the AI ​​model, or it can refer to a part of the computational task of the AI ​​model. For example, the data related to the first part can include the first part itself, or it can include the data generated by the computing chip during the processing of the first part by the first operator.

[0078] When the device includes only one chipset, the chipset is obtained by bonding computing chips and memory chips. When the device includes multiple chipsets, some or all of the chipsets are obtained by bonding computing chips and memory chips. Each chipset can be referred to as a process-in-memory (PIM) die. This is because the computing chips in the chipset have computing capabilities, and the memory chips in the chipset have storage capabilities; therefore, the computational tasks of AI models can be processed within the chipset through the computing chips and memory chips.

[0079] Because the chipset is formed by bonding computing chips and memory chips, there is no need to modify the computing chip internally, thus avoiding increasing its area and sacrificing computing power to maintain the same chip area. Similarly, there is no need to modify the memory chip internally, avoiding increasing its area and sacrificing its capacity and internal bandwidth to maintain the same chip area. The memory chip comprises multiple memory banks, each with its own storage capacity. Each bank can perform independent read and write operations, and the capacity and internal bandwidth of the memory chip are directly proportional to the number of banks.

[0080] Furthermore, in this embodiment, the computing chip and the memory chip can be bonded after they have been manufactured separately. Therefore, during the manufacturing process, the computing chip and the memory chip can each employ suitable processes, enabling the computing chip to possess strong computing power and the memory chip to have large capacity and internal bandwidth. This ensures that the chipset obtained by bonding the computing chip and the memory chip meets practical requirements in terms of computing power, capacity, and internal bandwidth. In some embodiments, the process technology of the computing chip differs from that of the memory chip. In other embodiments, the process technology of the computing chip is the same as that of the memory chip.

[0081] This application does not limit the type of computing chip or the type of storage chip. In one example, the computing chip is a complementary metal-oxide-semiconductor (CMOS) circuit chip, and the storage chip is a dynamic random access memory (DRAM) chip. Of course, the storage chip can also be a storage class memory (SCM) chip other than a DRAM chip, and can be selected based on experience or actual needs.

[0082] For example, the bonding method between the computing chip and the memory chip is hybrid bonding, which is used for three-dimensional (3D) packaging of the computing chip and the memory chip. For example, the computing chip includes a second interface, and the memory chip includes a third interface. The computing chip and the memory chip are bonded through the second interface and the third interface. For instance, as shown in Figure 2, the computing chip includes an input / output (I / O) interface 1 (i.e., the second interface) and multiple computing circuits, while the memory chip includes an I / O interface 2 (i.e., the third interface) and multiple banks. Hybrid bonding is used to bond the I / O interface 1 of the computing chip and the I / O interface 2 of the memory chip, forming multiple circuits between I / O interface 1 and I / O interface 2, thereby achieving the bonding between the computing chip and the memory chip. These multiple circuits are internal circuits of the chipset, used for data exchange between the computing chip and the memory chip to enable the computing chip and the memory chip to process AI model computation tasks. For example, the computing chip acquires data stored in the memory chip and performs calculations based on the acquired data to process the AI ​​model computation tasks. The number of circuits can be set according to actual needs, such as the number of banks included in the memory chip, which helps to improve the internal bandwidth of the memory chip. The theoretical value of the internal bandwidth of the memory chip is the product of the number of circuits and the clock frequency of the memory chip.

[0083] Because hybrid bonding enables 3D packaging of both the computing chip and the memory chip, it eliminates the need for internal modifications within either chip. This approach balances the area and computing power of the computing chip with the area, capacity, and internal bandwidth of the memory chip. Furthermore, in this embodiment, the computing chip and memory chip can be manufactured separately before being 3D packaged using hybrid bonding. Consequently, each chip can employ appropriate manufacturing processes, ensuring that the chipset's computing power, capacity, and bandwidth meet practical requirements.

[0084] For example, referring to Figure 1, embodiments of this application can package at least one chipset to obtain a PIM chip, which can also be called a package. For example, a PIM chip may include 2 chipsets, 4 chipsets, etc. Embodiments of this application do not limit the number of chipsets included in a PIM chip. Different PIM chips may include the same number of chipsets or different numbers of chipsets.

[0085] In an exemplary embodiment, referring to FIG3, the device further includes a first interface and a control chip (also referred to as a controller). Exemplarily, the first interface, the control chip, and at least one chipset are respectively located on a baseboard. The first interface is used for the control chip to connect to a processing chip outside the device. The first interface, the control chip, at least one chipset, and the baseboard constitute a PIM disk. Since at least one chipset can be packaged as a PIM chip, the PIM disk can also be considered to include the first interface, the control chip, at least one PIM chip, and the baseboard. For example, referring to FIG4, FIG4 shows a PIM disk including a first interface, a control chip, multiple PIM chips, and a baseboard. For example, the multiple PIM chips are PIM chip 0 to PIM chip 31 shown in FIG4, a total of 32 PIM chips. The processing chip is used to send the first part of the AI ​​model's computational task to the control chip through the first interface. The control chip is used to receive the first part sent by the processing chip through the first interface. The control chip is also used to obtain the processing result of the first part and send the processing result of the first part to the processing chip through the first interface.

[0086] In one example, the first part refers to all the computational tasks of the AI ​​model; that is, the computational tasks of the AI ​​model only include the first part. In this example, the processing chip offloads all operators required to process the computational tasks of the AI ​​model to the PIM disk, so that the PIM disk can handle all the computational tasks. For example, all operators can be offloaded to the computing chip in the PIM disk. Or, all operators can be offloaded to the computing chip and control chip in the PIM.

[0087] In another example, the first part refers to a portion of the computational tasks of the AI ​​model; that is, the computational tasks of the AI ​​model include both a first part and a second part. In this example, the processing chip offloads a portion of the operators required to process the computational tasks of the AI ​​model to the PIM disk (offloaded to the computing chip, or, offloaded to both the computing chip and the control chip), so that the PIM disk processes the first part, while the other portion of the operators remains on the processing chip, so that the processing chip processes the second part.

[0088] Optionally, considering that the PIM disk can provide high bandwidth and storage capacity, some operators offloaded to the PIM disk can be operators with lower computing power requirements but higher bandwidth and storage capacity requirements. Considering that the processing chip can provide high computing power, other operators retained in the processing chip can be operators with higher computing power requirements but lower bandwidth and storage capacity requirements.

[0089] For example, the computational tasks of an AI model include training and inference tasks. Operators used for training tasks have high computational power requirements but low requirements for bandwidth and storage capacity. Offloading them to the PIM disk yields minimal benefits (making them unsuitable for PIM disk offloading), so they can remain on the processing chip. Operators used for inference tasks have lower computational power requirements but higher requirements for bandwidth and storage capacity. Offloading them to the PIM disk yields greater benefits (making them suitable for PIM disk offloading), so they can be offloaded to the PIM disk.

[0090] For example, inference tasks include key-value (KV) caching tasks, numerical computation tasks, and other tasks. Operators used for other tasks have higher computational power requirements and lower requirements for bandwidth and storage capacity, so they can remain on the processing chip. Operators used for numerical computation tasks have moderate computational power requirements and moderate requirements for bandwidth and storage capacity, so they can remain on the processing chip or be offloaded to the PIM disk. Operators used for key-value caching tasks have lower computational power requirements and higher requirements for bandwidth and storage capacity, so they can be offloaded to the PIM disk. If both operators used for numerical computation tasks and operators used for key-value caching tasks are offloaded to the PIM disk, in one example, the operators used for numerical computation tasks can be offloaded to the control chip on the PIM disk, and the operators used for key-value caching tasks can be offloaded to the computing chip on the PIM disk.

[0091] Of course, the above operator unloading methods are merely examples and are not intended to limit the embodiments of this application.

[0092] For example, when there are multiple chipsets in the PIM disk, the arrangement of the multiple chipsets on the baseboard can be, but is not limited to, the following two.

[0093] The first arrangement involves multiple chipsets arranged side-by-side on the baseplate. In other words, multiple chipsets are laid flat on the baseplate. This arrangement is relatively simple and easy to implement. See Figure 5, which illustrates the case where multiple chipsets (i.e., chipset 0 to chipset N, where N is a positive integer greater than or equal to 1) are arranged side-by-side on the baseplate.

[0094] The second arrangement involves stacking at least two chipsets on the backplane. With a fixed backplane area, stacking the chipsets allows for a larger number of chipsets on the backplane, providing greater computing power, capacity, and internal bandwidth. See Figure 6, which illustrates the case where at least two chipsets (i.e., chipset 0 to chipset N) are stacked on the backplane.

[0095] Of course, the embodiments of this application can combine the first arrangement method and the second arrangement method. For example, a portion of the chipsets on the base plate are arranged side by side according to the first arrangement method, while another portion of the chipsets on the base plate are stacked according to the second arrangement method. Regardless of the arrangement method used, for a single chipset, the chipset can be arranged with the memory chip on top and the computing chip on the bottom (i.e., the distance between the memory chip and the base plate is greater than the distance between the computing chip and the base plate), or the chipset can be arranged with the computing chip on top and the memory chip on the bottom (i.e., the distance between the computing chip and the base plate is greater than the distance between the memory chip and the base plate).

[0096] For example, the processing chips mentioned above include, but are not limited to, central processing units (CPUs), graphics processing units (GPUs), tensor processing units (TPUs), neural network processing units (NPUs), and so on. The processing chip may include high bandwidth memory (HBM), a type of high-performance DRAM chip. Since the control chip needs to receive the first part of the AI ​​model's computational task sent by the processing chip, the processing chip plays a leading role in processing the AI ​​model's computational task; therefore, the processing chip can also be called the main chip.

[0097] This application does not limit the relative positional relationship between the control chip and at least one chipset on the baseboard. For example, any two components from the control chip and at least one chipset can be located on the same surface of the baseboard or on different surfaces. Any two components can refer to at least one of the following: one control chip and one chipset, two control chips, or two chipsets. Different surfaces can refer to two adjacent surfaces on the baseboard, or two opposite surfaces on the baseboard.

[0098] In some implementations, the control chip and chipset are electrically connected via a backplane. For example, the backplane is a printed circuit board (PCB) with printed circuitry, and the control chip and chipset are electrically connected via this printed circuitry. In other implementations, the backplane only serves to house the control chip and chipset, which are electrically connected via other means outside the backplane. For example, the control chip and chipset are electrically connected via circuitry outside the backplane.

[0099] This application does not limit the electrical connection method between the control chip and the chipset. The electrical connection between the control chip and the chipset is used for data exchange between them, such as for the control chip sending a first part of a computational task to the chipset, and for the control chip obtaining the processing result of the first part from the chipset. Exemplarily, the protocols used in the data exchange between the control chip and the chipset include, but are not limited to, double data rate (DDR) protocol, or a first proprietary protocol, etc. When the protocol is DDR, the circuit between the control chip and the chipset is a DDR bus. When the protocol is a first proprietary protocol, the circuit between the control chip and the chipset is a proprietary bus.

[0100] Furthermore, the control chip and the processing chip are connected via a first interface for data exchange between them. This interface is used for the control chip to obtain a first part of a computational task from the processing chip, and for the control chip to send the processing result of the first part to the processing chip. Exemplarily, the protocols used in the data exchange between the control chip and the processing chip include, but are not limited to: the Compute Express Link (CXL) protocol, the Peripheral Component Interconnect Express (PCIe) protocol, or a second proprietary protocol, etc. The second proprietary protocol may be the same as or different from the first proprietary protocol described above; this application does not limit this. When the protocol is CXL, the circuit containing the first interface between the control chip and the processing chip is a CXL bus. When the protocol is PCIe, the circuit containing the first interface between the control chip and the processing chip is a PCIe bus. When the protocol is the second proprietary protocol, the circuit containing the first interface between the control chip and the chipset is a proprietary bus.

[0101] In this embodiment, a processing chip can be connected to at least one AI model computing task processing device (PIM disk). This embodiment does not limit the number of PIM disks connected to each processing chip. Different processing chips can be connected to the same or different numbers of PIM disks. For example, referring to Figure 7, each of the multiple processing chips (i.e., processing chip 0 to processing chip M, where M is a positive integer greater than or equal to 1, for example, M is 8) is connected to two PIM disks.

[0102] For example, as shown in Figure 8, the processing chip can be located in the inference server. The inference server includes a fourth interface for connecting to the processing chip and also for connecting to the PIM disk. Optionally, each processing chip can correspond to at least one fourth interface. By inserting the PIM disk into the fourth interface corresponding to a processing chip, the PIM disk can be connected to that processing chip. Accordingly, the control chip in the PIM disk can receive the first part of the computational task sent by the processing chip as described above and send it to the chipset in the PIM disk, enabling the chipset to perform the processing of the first part. In one example, the AI ​​model training task is completed by the processing chip. After the training process is completed, the PIM disk is inserted, and the entire inference task of the AI ​​model is completed by the PIM disk. Alternatively, the processing chip and the PIM disk can each complete a portion of the AI ​​model's inference task.

[0103] The preceding text primarily described the structure of the computational task processing device for the AI ​​model. The following section details the processing procedure for the computational task. For example, the processing procedure for the computational task includes, but is not limited to, the following two scenarios.

[0104] In the first scenario, the first part of the AI ​​model's computational task is a partial computational task, meaning the AI ​​model's computational task includes both a first part and a second part. Correspondingly, the processing chip also includes a second operator, which is used to process the second part of the AI ​​model's computational task. In other words, in the first scenario, the PIM disk needs to collaborate with the processing chip to complete the AI ​​model's computational task. The PIM disk completes the first part of the computational task, and the processing chip completes the second part. Essentially, the processing chip offloads some of the operators required for processing the AI ​​model's computational task to the PIM disk, while retaining the other part. For example, referring to Figure 7, the PIM disk and the processing chip each complete a portion of the computational task involved in the AI ​​model's inference process.

[0105] In an exemplary embodiment, the PIM disk completes the first part of the computing task in the following ways, including but not limited to the following two.

[0106] In the first processing method, the first part includes a first sub-part and a second sub-part, and the control chip includes a third operator. A computing chip processes the first sub-part using the first operator. The control chip also processes the second sub-part using the third operator. The control chip obtains the processing result of the first part based on the processing results of the first and second sub-parts. In other words, in the first processing method, the computing chip and the control chip in the chipset jointly complete the processing of the first part to obtain its result. This first method is advantageous for improving the processing efficiency of the first part.

[0107] For example, the first sub-part includes a first calculation instruction and first data to be calculated. Processing the first sub-part means: performing calculations on the first data using a first operator according to the calculation method indicated by the first calculation instruction to obtain a first processing result. The second sub-part includes a second calculation instruction and second data to be calculated. Processing the second sub-part means: performing calculations on the second data using a third operator according to the calculation method indicated by the second calculation instruction to obtain a second processing result. The embodiments of this application do not limit the processing order of the first and second sub-parts.

[0108] For example, the processing order of the first sub-part is earlier than the processing order of the second sub-part. After the control chip receives the first part sent by the processing chip through the first interface, it first sends the first sub-part to the chipset. The chipset processes the first sub-part through the first operator to obtain the first processing result, and then sends the first processing result back to the control chip. Afterwards, based on the first processing result, the control chip processes the second sub-part through the third operator to obtain the second processing result. This second processing result is the processing result of the first part sent by the control chip to the processing chip.

[0109] For example, the processing order of the first sub-part may be later than that of the second sub-part. After the control chip receives the first part sent by the processing chip through the first interface, it first processes the second sub-part through the third operator to obtain the second processing result, and then sends the first sub-part to the chipset. Based on the second processing result, the chipset processes the first sub-part through the first operator to obtain the first processing result, and sends the first processing result to the control chip. This first processing result is the processing result of the first part sent by the control chip to the processing chip.

[0110] For example, the processing order of the first sub-part is the same as that of the second sub-part. After the control chip receives the first part sent by the processing chip through the first interface, it sends the first sub-part to the chipset. The chipset processes the first sub-part through the first operator to obtain the first processing result, and sends the first processing result back to the control chip. Furthermore, the control chip processes the second sub-part through the third operator to obtain the second processing result. Therefore, the processing result of the first part sent by the control chip to the processing chip includes both the first and second processing results.

[0111] In the second processing method, the chipset independently completes the processing of the first part.

[0112] For example, after the control chip receives the first part sent by the processing chip through the first interface, it sends the first part to the chipset. The chipset processes the first part through the first operator, obtains the processing result of the first part, and sends the processing result of the first part to the control chip. Then, the control chip sends the processing result of the first part to the processing chip.

[0113] For example, in either the first or second processing method, the control chip can be used to manage the chipset. For instance, the control chip sends a read command to the chipset to read data stored in the memory chips within the chipset. Alternatively, the control chip sends a write command to the chipset to write data to the memory chips within the chipset. Furthermore, the memory chips in the chipset have corresponding processing queues, which store at least one of the read or write commands. The control chip manages these processing queues, for example, by adjusting the order of at least one of the commands included in the processing queue, or by prompting the memory chips to process at least one command in the processing queue.

[0114] Furthermore, the processing order of the first part and the second part is not limited in the embodiments of this application. For example, the processing order of the first part is earlier than the processing order of the second part. After the control chip receives the first part sent by the processing chip through the first interface, the PIM disk processes the first part, and then the control chip sends the processing result of the first part to the processing chip. Then, based on the processing result of the first part, the processing chip processes the second part through the second operator to obtain the processing result of the second part, which is the processing result of the AI ​​model's calculation task. Alternatively, the processing order of the first part is later than the processing order of the second part. The processing chip first processes the second part through the second operator to obtain the processing result of the second part, and then sends the first part to the control chip. The PIM disk processes the first part based on the processing result of the second part, and then the control chip sends the processing result of the first part to the processing chip, which is the processing result of the AI ​​model's calculation task. Finally, the processing order of the first part is the same as the processing order of the second part. The control chip receives the first part sent by the processing chip through the first interface. The PIM disk processes the first part, and then the control chip sends the processing result of the first part to the processing chip. Furthermore, the processing chip processes the second part through the second operator to obtain the processing result of the second part. The processing result of the AI ​​model's computation task includes the processing results of the first and second parts.

[0115] Since in the first scenario, the PIM disk is used to complete the first part of the computing task, and the processing chip is used to complete the second part, this embodiment of the application needs to reasonably allocate the first part and the second part to the PIM disk and the processing chip respectively, so as to make full use of the PIM disk and the processing chip, thereby improving the processing efficiency of the computing task. Exemplarily, the allocation method in this embodiment of the application includes, but is not limited to, at least one of the following.

[0116] In the first allocation method, the computing power provided by the control chip and chipset is less than that provided by the processing chip, and the computing power consumed by the first part is less than that consumed by the second part. For example, the computing power requirement of the computing chip in the chipset, including the first operator, is lower than the computing power requirement of the second operator included in the processing chip. Optionally, the computing power requirement of the third operator included in the control chip is also lower than the computing power requirement of the second operator.

[0117] Due to manufacturing processes and other reasons, the computing power provided by the control chip and chipset, i.e., the computing power provided by the PIM disk, may be less than that provided by the processing chip. If the first part allocated to the PIM disk requires more computing power, the PIM disk will need to spend a longer time processing the first part, reducing the processing efficiency of the first part and also reducing the processing efficiency of the computing task. Therefore, the embodiments of this application ensure that the first part allocated to the PIM disk requires only less computing power, i.e., the computing power consumed by the first part is less than that consumed by the second part. Thus, the PIM disk and the processing chip can each efficiently complete the processing of the first and second parts, thereby ensuring the processing efficiency of the computing task.

[0118] The second allocation method is where the sum of the data volume of the first part and the processing results of the first part is less than the data volume threshold.

[0119] The first part of the data volume refers to the amount of data that the control chip needs to receive from the processing chip through the first interface to perform processing. The data volume of the processing result of the first part is the amount of data that the control chip sends to the processing chip through the first interface. Therefore, the sum of the data volumes of the first part and the processing result of the first part is the total amount of data exchanged between the control chip and the processing chip through the first interface, which is also the total amount of data exchanged between the PIM disk and the main chip.

[0120] If the total data volume exceeds the data volume threshold, it indicates that the control chip and processing chip need to exchange a large amount of data, resulting in a high bandwidth utilization rate for the first interface. Considering that the bandwidth of this first interface is less than the internal bandwidth of the chipset, for example, 1 / 10 or 1 / 100 of the internal bandwidth of the chipset, a high bandwidth utilization rate for this first interface may make it a bottleneck for the computing task. For example, the control chip may need to wait a relatively long first time to obtain the first part from the first interface, which is longer than the second time required for the PIM disk to process the first part. Similarly, the processing chip may need to wait a relatively long third time to obtain the processing result of the first part from the first interface, which is longer than the fourth time required for the processing chip to process the second part. For example, Figure 9 illustrates an exemplary proportional relationship between the first, second, third, and fourth times.

[0121] Therefore, in this embodiment of the application, the sum of the above data amounts is less than the data amount threshold, so that the control chip and the processing chip only need to exchange a small amount of data, avoiding excessive bandwidth occupancy of the first interface and ensuring the processing efficiency of the computing task.

[0122] In the third allocation method, the bandwidth provided by the control chip and chipset is greater than the bandwidth provided by the processing chip, and the bandwidth consumed by the first part is greater than the bandwidth consumed by the second part. For example, the bandwidth requirement of the computing chip in the chipset, including the first operator, is higher than the bandwidth requirement of the processing chip, including the second operator. Optionally, the bandwidth requirement of the control chip, including the third operator, is also higher than the bandwidth requirement of the second operator.

[0123] The bandwidth provided by the control chip and chipset, i.e., the bandwidth provided by the PIM disk, may be greater than the bandwidth provided by the processing chip. Optionally, the bandwidth provided by the PIM disk is the internal bandwidth of the memory chip in the chipset, and the bandwidth provided by the processing chip is the internal bandwidth of the HBM. If the second part allocated to the processing chip requires more bandwidth, the processing chip will need to spend a longer time to complete the processing of the second part, reducing the processing efficiency of the second part and also reducing the processing efficiency of the computing task. Therefore, the embodiments of this application ensure that the second part allocated to the processing chip only requires less bandwidth, i.e., the bandwidth consumed by the first part is greater than the bandwidth consumed by the second part. Thus, the PIM disk and the processing chip can each efficiently complete the processing of the first and second parts, ensuring the processing efficiency of the computing task.

[0124] The third allocation method is merely an example and is not intended to limit the allocation methods of the first and second parts. For instance, the bandwidth provided by the control chip and chipset can be less than the bandwidth provided by the processing chip. Correspondingly, the bandwidth consumed by the first part can also be less than the bandwidth consumed by the second part. This situation can also ensure the processing efficiency of the computing task.

[0125] The fourth allocation method involves the control chip and chipset providing more capacity than the processing chip, and the first part consuming more capacity than the second part. For example, the computing chip in the chipset, including the first operator, has a higher capacity requirement than the processing chip, including the second operator. Optionally, the control chip, including the third operator, also has a higher capacity requirement than the capacity operator's bandwidth requirement.

[0126] The capacity provided by the control chip and chipset, i.e., the capacity provided by the PIM disk, may be greater than the capacity provided by the processing chip. Optionally, the capacity provided by the PIM disk is the capacity of the storage chips in the chipset, and the capacity provided by the processing chip is the capacity of the HBM. If the second part allocated to the processing chip requires more capacity, the processing chip will need to spend a longer time to complete the processing of the second part, reducing the processing efficiency of the second part and also reducing the processing efficiency of the computing task. Therefore, the embodiments of this application ensure that the second part allocated to the processing chip only requires less capacity, i.e., the capacity consumed by the first part is greater than the capacity consumed by the second part. Thus, the PIM disk and the processing chip can each efficiently complete the processing of the first and second parts, ensuring the processing efficiency of the computing task.

[0127] The fourth allocation method is merely an example and is not intended to limit the allocation methods of the first and second parts. For instance, the capacity provided by the control chip and chipset can be less than the capacity provided by the processing chip. Correspondingly, the capacity consumed by the first part can also be less than the capacity consumed by the second part. This situation can also ensure the processing efficiency of the computing task.

[0128] This application embodiment can combine the above allocation methods. For example, the first part and the second part satisfy all of the following conditions: the computing power consumed by the first part is less than that consumed by the second part, the bandwidth consumed by the first part is greater than that consumed by the second part, the capacity consumed by the first part is greater than that consumed by the second part, and the sum of the data volume corresponding to the first part and the data volume of the processing result is less than the data volume threshold. That is, a portion of the operators (including the first operator and the third operator) offloaded to the PIM disk are operators with lower computing power requirements and higher bandwidth and storage capacity requirements, while the other portion of the operators (including the second operator) retained in the processing chip are operators with higher computing power requirements and lower bandwidth and storage capacity requirements. Another example is that the first part and the second part satisfy all of the following conditions: the computing power consumed by the first part is less than that consumed by the second part, the bandwidth consumed by the first part is less than that consumed by the second part, the capacity consumed by the first part is less than that consumed by the second part, and the sum of the data volume corresponding to the first part and the data volume of the processing result is less than the data volume threshold. In other words, some operators (including the first and third operators) that are offloaded to the PIM disk have lower requirements for computing power, bandwidth, and storage capacity, while other operators (including the second operator) that are retained in the processing chip have higher requirements for computing power, bandwidth, and storage capacity.

[0129] Next, using the generative pre-trained transformer (GPT) model as an example, we will illustrate the first and second parts. The GPT model is a neural network model that uses a transformer architecture, including but not limited to chat GPT models. GPT models can generate content such as text, images, or music based on user input and output this content to the user in the form of tokens. For example, if the content is text, the token could be a single character or a phrase. GPT models can be used in various scenarios such as question answering, text summarization, content generation, and search.

[0130] The computational tasks involved in the GPT model (e.g., those involved in the inference process of the GPT model) include, but are not limited to: general matrix-vector multiplication (GEMV), general matrix multiplication (GEMM), or all reduce, etc. As shown in Figure 10, GEMV is used to calculate the product between a matrix and a vector, and GEMM is used to calculate the product between different matrices. All reduce is used to sum the computational results from various computing platforms. For example, for a control chip, each computing platform can refer to a chipset. The control chip can sum the computational results from each chipset (such as the first processing result described above) to obtain a summed result (such as the processing result in the first part described above). The control chip can also send the summed result to the processing chip.

[0131] Referring to Figure 11, in one example, GEMM requires more computing power and has a larger number of operations, but it consumes less bandwidth and capacity, resulting in a smaller latency. GEMV, on the other hand, requires less computing power and has a smaller number of operations, but it consumes more bandwidth and capacity, resulting in a larger latency. Therefore, in this example, GEMV can be considered the first part, handled by the PIM disk, while GEMM can be considered the second part, handled by the processing chip.

[0132] When a GPT model includes a large number of model parameters, it is also called a foundation model or a large model. The inference process of the foundation model has high real-time requirements; for example, it needs to output a token to the user every 50 milliseconds to ensure a real-time user experience. If all computational tasks in the inference process of the foundation model are completed solely by the processing chip, the bandwidth and capacity provided by the processing chip will be insufficient due to the area constraints of the processing chip and the cost of HBM, resulting in low processing efficiency, increased processing latency, and difficulty in meeting real-time requirements. Furthermore, insufficient bandwidth and capacity provided by the processing chip will also constrain the batch processing capability. For example, due to insufficient bandwidth and capacity, the processing chip can only handle a small number of concurrent computational tasks. To address this, the method provided in this application embodiment can be used to enable the PIM disk and the processing chip to collaborate in completing computational tasks. This is equivalent to increasing the bandwidth and capacity through the PIM disk, thereby improving the processing efficiency of computational tasks and enhancing the ability to handle concurrent computational tasks. For example, through the collaboration between the PIM disk and the processing chip, dozens or even hundreds of concurrent computing tasks can be processed. These computing tasks can be long sequence tasks, which refer to computing tasks with sequence lengths of hundreds of kilobytes or more.

[0133] In an exemplary embodiment, the computational task of the AI ​​model is an inference task. Referring to Figure 12 and the above description, the inference task can include: key-value caching tasks, numerical computation tasks, and other tasks. Numerical computation tasks include, but are not limited to, at least one of the following: masking tasks, softmax normalization tasks, or scaling tasks. The masking task is used to set some elements of a matrix to 0, the softmax task is used to perform exponential operations on the elements of a matrix, and the scaling task is used to perform division operations on the elements of a matrix. Other tasks include, but are not limited to, at least one of the following: layer normalization tasks, Q-vector generation tasks, K-vector generation tasks, V-vector generation tasks, projection tasks, etc. Based on this, the first and second parts include at least the following allocation scenarios.

[0134] In the first allocation scenario, the first part consists of key-value caching tasks and numerical computation tasks, while the second part differs from both. For example, the second part might consist of the other tasks mentioned above.

[0135] The embodiments of this application do not limit the processing of the first part by the control chip and the chipset. For example, according to the first processing method described above, the control chip and the chipset can jointly complete the key-value caching task and the numerical calculation task. Alternatively, according to the second processing method described above, the chipset can independently complete the key-value caching task and the numerical calculation task.

[0136] For example, in the first processing method described above, the first sub-part of the first part is a key-value caching task, which is handled by the chipset; the second sub-part of the first part is a numerical calculation task, which is handled by the control chip. This task allocation is adopted because the numerical calculation task may require data stored in memory chips across multiple chipsets. If the chipset handles the numerical calculation task, data transfer between different chipsets may be introduced, potentially affecting the processing efficiency of the first part. Therefore, by having the control chip handle the numerical calculation task, data transfer between different chipsets is eliminated; only data transfer between the control chip and the chipset is required.

[0137] For example, referring to Figure 12, the processing chip first processes the normalization task based on the user's input, then processes the Q-vector generation task, K-vector generation task, and V-vector generation task to obtain the Q-vector, K-vector, and V-vector, and sends the Q-vector, K-vector, and V-vector to the control chip in the PIM disk. The control chip then sends the Q-vector, K-vector, and V-vector to the chipset. The chipset stores the K-vector through the storage chip and calculates the transpose (K-vector) of the Q-vector and the K-vector through the computing chip. T The first product QK between ) T Then, the chipset sends the first product QK to the control chip in the PIM disk. T The control chip is based on the first product QK T Process the mask task, softmax task, and scale task to obtain the result MQK. T Send MQK to the chipset T Additionally, the chipset stores the V vector via a memory chip, which is then used when the MQK is received. T Then, the MQK is calculated using a computing chip. T The second product Y between the vector V and the control chip is sent to the processing chip. The processing chip then processes the mapping task based on the second product Y and performs subsequent processing to obtain the token to be output to the user.

[0138] The second allocation scenario involves a first part consisting of key-value caching tasks, while the second part differs from key-value caching tasks. For example, the second part might consist of numerical computation tasks and other tasks.

[0139] The embodiments of this application do not limit the processing of the first part by the control chip and the chipset. For example, the key-value caching task can be completed jointly by the control chip and the chipset according to the first processing method described above. Or, according to the second processing method described above, the key-value caching task can be completed by the chipset alone.

[0140] In the second scenario, the control chip obtains all computational tasks from the processing chip via the first interface. The chipset processes all computational tasks to obtain a first processing result. The control chip also obtains the first processing result and sends a first portion of the processing result to the processing chip via the first interface. The first portion of the processing result is determined based on the first processing result. In other words, in the second scenario, the computational tasks obtained by the control chip constitute the entire computational task. The PIM disk does not need to collaborate with the processing chip; the PIM disk independently completes all computational tasks, while the processing chip does not need to complete any computational tasks. For example, referring to Figure 7, the PIM disk completes all computational tasks involved in the inference process of the AI ​​model.

[0141] In the second scenario, the first part of the AI ​​model's computational task constitutes the entire computational task of the AI ​​model; that is, the AI ​​model's computational task only includes the first part. In other words, in the second scenario, the PIM disk does not need to collaborate with the processing chip; the PIM disk can independently complete the AI ​​model's computational task. This is equivalent to the processing chip offloading all the operators required to process the AI ​​model's computational task to the PIM disk.

[0142] For example, the control chip and chipset can refer to the first processing method described above to jointly handle all computational tasks. In one example, all computational tasks include key-value caching tasks, numerical computation tasks, and other tasks. The chipset is used to handle key-value caching tasks, and the control chip is used to handle numerical computation tasks and other tasks.

[0143] Alternatively, the control chip and chipset can follow the second approach described above, with the chipset independently completing all computational tasks. In one example, the chipset is used to handle key-value caching tasks, numerical computation tasks, and other tasks.

[0144] Regarding the first and second scenarios described above, the processing chip can identify the first operator included in the computing chip of the PIM disk, obtain the identification result, and the identification result indicates the computing tasks supported by the first operator. The processing chip can then send a first part to the control chip based on the identification result. This first part represents the computing tasks supported by the first operator in the AI ​​model's computing tasks. For example, this first part could be a portion of the computing tasks in the first scenario or all of the computing tasks in the second scenario. In other words, the processing chip is also used to identify the first operator, obtain the identification result, and send the first part to the control chip based on the identification result.

[0145] The computational task processing apparatus for the AI ​​model provided in the embodiments of this application has been described above. Next, the computational task processing apparatus for this AI model will be compared and described with apparatuses in related technologies.

[0146] Referring to Figure 13, in related technologies, a processing unit (PU) is added next to each bank inside the DRAM chip. The PU is used to handle computational tasks such as multiply-accumulate (MAC) or activation function (AF). However, related technologies have at least the following technical problems.

[0147] The first technical issue is that, since the relevant technology involves improvements made inside the DRAM chip, it either increases the area of ​​the DRAM chip or, while keeping the area of ​​the DRAM chip unchanged, reduces the number of banks, thus affecting the capacity and internal bandwidth of the DRAM chip.

[0148] The second technical issue is that since both the bank and the PU are located inside the DRAM chip, the same process can only be used for the bank and the PU due to limitations in the manufacturing process. This may result in a smaller capacity and internal bandwidth for the bank, and a weaker computing power for the PU.

[0149] The third technical problem is that the DRAM chips in the relevant technologies are only used for single computing tasks and have poor scalability.

[0150] The technical problems that can be overcome by the embodiments of this application include at least the three technical problems mentioned above.

[0151] To address the first technical problem, the embodiments of this application bond the computing chip and the storage chip to obtain a chipset, without the need for modifications inside the computing chip or the storage chip. Therefore, it can balance the area and computing power of the computing chip, as well as the area, capacity and internal bandwidth of the storage chip.

[0152] Regarding the second technical problem, the embodiments of this application can bond the computing chip and the memory chip after they have been manufactured separately, so that the computing chip and the memory chip can each adopt appropriate processes, so that the computing chip has strong computing power and the memory chip has large capacity and bandwidth.

[0153] Regarding the third technical issue, the device (PIM disk) in this application embodiment can collaborate with the processing chip, and according to actual needs, one processing chip can collaborate with multiple PIM disks, exhibiting strong scalability. Furthermore, for the case of collaboration between the processing chip and the PIM disk, this application embodiment can also rationally allocate the tasks processed by the processing chip and the PIM disk based on at least one of the following indicators: computing power, capacity, bandwidth required for the computing task, or the amount of data exchanged between the processing chip and the PIM disk. This fully utilizes the capabilities of both the processing chip and the PIM disk, ensuring the efficiency of each processing chip and the PIM disk in handling computing tasks and reducing processing latency.

[0154] In summary, since the chipset is obtained by bonding the computing chip and the memory chip in this embodiment, no internal modifications to the computing chip are required, the area of ​​the computing chip is not increased, and the computing power of the computing chip is not sacrificed to maintain the same area. Similarly, no internal modifications to the memory chip are required, the area of ​​the memory chip is not increased, and the capacity and bandwidth of the memory chip are not sacrificed to maintain the same area. Furthermore, the bonding of the computing chip and the memory chip can be performed after they have been manufactured separately, allowing each chip to employ suitable processes. This results in the computing chip possessing strong computing power, and the memory chip possessing large capacity and bandwidth, thus ensuring that the chipset obtained by bonding the computing chip and the memory chip meets practical requirements in terms of computing power, capacity, and bandwidth.

[0155] The above describes the AI ​​model computation task processing apparatus provided in the embodiments of this application. In addition, the embodiments of this application also provide an inference server, which includes at least one AI model computation task processing apparatus. The apparatus includes at least one chipset, which is obtained by bonding a computing chip and a storage chip. The computing chip includes a first operator. The computing chip is used to process a first part of the AI ​​model computation task through the first operator. The storage chip is used to store data related to the first part.

[0156] Optionally, when the inference server includes a single device, that device is, for example, the device provided in the embodiments of this application described above. Alternatively, when the inference server includes multiple devices, at least one of the multiple devices is the device provided in the embodiments of this application described above.

[0157] In an exemplary embodiment, the apparatus further includes a first interface and a control chip. The control chip is configured to receive a first portion sent by the processing chip via the first interface. The control chip is also configured to acquire a processing result of the first portion and send the processing result of the first portion to the processing chip via the first interface.

[0158] For example, the processing chip is also used to identify the first operator, obtain the identification result, and send the first part to the control chip based on the identification result.

[0159] For example, the processing chip includes a second operator. The processing chip is also used to process a second part of the computational task of the AI ​​model through the second operator.

[0160] In one example, the storage capacity requirement of the first operator is higher than that of the second operator.

[0161] In another example, the bandwidth requirement of the first operator is higher than that of the second operator.

[0162] In yet another example, the computational power requirement of the first operator is lower than that of the second operator.

[0163] In another example, the sum of the data volume of the first part and the processing results of the first part is less than the data volume threshold.

[0164] In an exemplary embodiment, the first part includes a first sub-part and a second sub-part, and the control chip includes a third operator. A computing chip is used to process the first sub-part via the first operator; the control chip is further used to process the second sub-part via the third operator; and the control chip is used to obtain a processing result for the first part based on the processing results of the first sub-part and the second sub-part, respectively.

[0165] For example, the computation task is an inference task. The first part consists of a key-value caching task and a numerical computation task, or the first part consists of a key-value caching task.

[0166] For example, the computation task is a reasoning task, the first sub-part is a key-value caching task, and the second sub-part is a numerical computation task.

[0167] Optionally, the computing chip further includes a second interface, and the storage chip includes a third interface, with the computing chip and the storage chip bonded to each other via the second interface and the third interface.

[0168] This application also provides a method for processing computational tasks of an AI model. This method can be applied to a device for processing computational tasks of an AI model. The device includes at least one chipset, which is obtained by bonding a computing chip and a storage chip. The computing chip includes a first operator. Optionally, the device may be, for example, the device provided in the embodiments of this application described above.

[0169] As shown in Figure 14, the method includes the following steps 1401 and 1402.

[0170] Step 1401: The first part of the computational task of the AI ​​model is processed by the first operator through the computing chip.

[0171] Step 1402: Store the data related to the first part using the storage chip.

[0172] In an exemplary embodiment, the device further includes a first interface and a control chip. Referring to FIG15, the method provided in this application embodiment further includes the following steps 1403 to 1405.

[0173] Step 1403: The control chip receives the first part sent by the processing chip through the first interface. Optionally, the control chip can send the first part to the chipset.

[0174] Step 1404: Obtain the processing results of the first part through the control chip.

[0175] Step 1405: The control chip sends the first part of the processing result to the processing chip through the first interface.

[0176] For example, the method further includes: identifying the first operator through the processing chip to obtain an identification result, and sending a first part to the control chip based on the identification result. That is, the first part received by the control chip is the identification result obtained by the processing chip based on the identification of the first operator and sent to the control chip.

[0177] For example, the processing chip includes a second operator. The method further includes: processing a second part of the computational task of the AI ​​model by the processing chip through the second operator. That is, the computational task of the AI ​​model includes a first part and a second part, the second part being processed by the processing chip through the second operator.

[0178] In one example, the storage capacity requirement of the first operator is higher than that of the second operator.

[0179] In another example, the bandwidth requirement of the first operator is higher than that of the second operator.

[0180] In yet another example, the computational power requirement of the first operator is lower than that of the second operator.

[0181] In another example, the sum of the data volume of the first part and the processing results of the first part is less than the data volume threshold.

[0182] In an exemplary embodiment, the first part includes a first sub-part and a second sub-part, and the control chip includes a third operator. Processing the first part of the computational task of the AI ​​model by the computing chip through the first operator includes: processing the first sub-part by the computing chip through the first operator. The method further includes: processing the second sub-part by the control chip through the third operator. Obtaining the processing result of the first part by the control chip includes: obtaining the processing result of the first part by the control chip based on the respective processing results of the first sub-part and the second sub-part.

[0183] For example, the computation task is an inference task. The first part consists of a key-value caching task and a numerical computation task, or the first part consists of a key-value caching task.

[0184] For example, the computation task is a reasoning task, the first sub-part is a key-value caching task, and the second sub-part is a numerical computation task.

[0185] In an exemplary embodiment, the computing chip further includes a second interface, and the storage chip includes a third interface, wherein the computing chip and the storage chip are bonded to the third interface via the second interface.

[0186] The implementation methods and technical effects of the above-mentioned inference server and AI model computation task processing methods can be found in the implementation methods and technical effects of the AI ​​model computation task processing devices shown in Figures 1 to 8, which will not be elaborated here.

[0187] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another.

[0188] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0189] In this application, the term "at least one" means one or more, and the term "multiple" means two or more; for example, multiple chipsets refer to two or more chipsets. The terms "system" and "network" are often used interchangeably herein.

[0190] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0191] It should also be understood that the term "...or at least one of..." as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "...or at least one of..." describes an association relationship between related objects, indicating that three relationships can exist. For example, "at least one of A or B" can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0192] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0193] The above description is merely an embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A computational task processing device for an AI model, characterized in that, The device includes at least one chipset, which is obtained by bonding a computing chip and a storage chip, and the computing chip includes a first operator. The computing chip is used to process a first part of the computing task of the artificial intelligence (AI) model through the first operator; The memory chip is used to store data related to the first part.

2. The apparatus according to claim 1, characterized in that, The device also includes a first interface and a control chip; The control chip is used to receive the first part sent by the processing chip through the first interface; The control chip is also used to acquire the processing result of the first part and send the processing result of the first part to the processing chip through the first interface.

3. The apparatus according to claim 2, characterized in that, The processing chip is also used to identify the first operator, obtain an identification result, and send the first part to the control chip based on the identification result.

4. The apparatus according to claim 2 or 3, characterized in that, The processing chip includes a second operator; The processing chip is also used to process a second part of the computational task of the AI ​​model through the second operator.

5. The apparatus according to claim 4, characterized in that, The computing power requirement of the first operator is lower than that of the second operator.

6. The apparatus according to any one of claims 2-5, characterized in that, The sum of the data volume of the first part and the processing result of the first part is less than the data volume threshold.

7. The apparatus according to any one of claims 2-6, characterized in that, The first part includes a first sub-part and a second sub-part, and the control chip includes a third operator; The computing chip is used to process the first sub-part through the first operator; The control chip is also used to process the second sub-part through the third operator; The control chip is used to obtain the processing result of the first part based on the processing results of the first sub-part and the second sub-part respectively.

8. The apparatus according to any one of claims 1-7, characterized in that, The computational task is a reasoning task; The first part is a key-value caching task and a numerical calculation task, or the first part is the key-value caching task.

9. The apparatus according to claim 7, characterized in that, The computational task is an inference task, the first sub-part is a key-value caching task, and the second sub-part is a numerical computation task.

10. The apparatus according to any one of claims 1-9, characterized in that, The computing chip further includes a second interface, and the storage chip includes a third interface. The computing chip and the storage chip are bonded to the third interface through the second interface.

11. An inference server, characterized in that, The inference server includes a computing task processing device for at least one AI model. The device includes at least one chipset, which is obtained by bonding a computing chip and a storage chip. The computing chip includes a first operator. The computing chip is used to process a first part of the computing task of the artificial intelligence (AI) model through the first operator; The memory chip is used to store data related to the first part.

12. The inference server according to claim 11, characterized in that, The device also includes a first interface and a control chip; The control chip is used to receive the first part sent by the processing chip through the first interface; The control chip is also used to acquire the processing result of the first part and send the processing result of the first part to the processing chip through the first interface.

13. The inference server according to claim 12, characterized in that, The processing chip is also used to identify the first operator, obtain an identification result, and send the first part to the control chip based on the identification result.

14. The inference server according to claim 12 or 13, characterized in that, The processing chip includes a second operator; The processing chip is also used to process a second part of the computational task of the AI ​​model through the second operator.

15. The inference server according to claim 14, characterized in that, The computing power requirement of the first operator is lower than that of the second operator.

16. The inference server according to any one of claims 12-15, characterized in that, The sum of the data volume of the first part and the processing result of the first part is less than the data volume threshold.

17. The inference server according to any one of claims 12-16, characterized in that, The first part includes a first sub-part and a second sub-part, and the control chip includes a third operator; The computing chip is used to process the first sub-part through the first operator; The control chip is also used to process the second sub-part through the third operator; The control chip is used to obtain the processing result of the first part based on the processing results of the first sub-part and the second sub-part respectively.

18. The inference server according to any one of claims 11-17, characterized in that, The computational task is a reasoning task; The first part is a key-value caching task and a numerical calculation task, or the first part is the key-value caching task.

19. The inference server according to claim 17, characterized in that, The computational task is an inference task, the first sub-part is a key-value caching task, and the second sub-part is a numerical computation task.

20. The inference server according to any one of claims 11-19, characterized in that, The computing chip further includes a second interface, and the storage chip includes a third interface. The computing chip and the storage chip are bonded to the third interface through the second interface.

Images