CPU, decoding method based on hybrid expert network and related products
By introducing an AI computing unit into the CPU, specifically designed for performing expert calculations, the problem of uneven computing resources in hybrid expert networks is solved, resulting in more efficient resource utilization and improved overall throughput.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING BAIDU NETCOM SCI & TECH CO LTD
- Filing Date
- 2025-12-23
- Publication Date
- 2026-05-05
AI Technical Summary
In existing technologies, the uneven utilization of computing resources in hybrid expert networks leads to limited overall throughput, especially in the decoding process of large language models, where the uneven allocation of resources between CPU and AI chips affects performance.
By introducing an AI computing functional unit into the CPU, which is dedicated to performing expert computing tasks, expert computing is offloaded from the AI chip to the CPU. Through the independent design of the general computing functional unit and the AI computing functional unit, balanced utilization of resources is achieved.
By balancing resource utilization, overall throughput and performance were improved, and the decoding efficiency of large language models was enhanced.
Smart Images
Figure CN121979666A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to the fields of cloud computing, large models, and computing power. Specifically, it relates to a CPU, a decoding method, apparatus, device, medium, and product based on a hybrid expert network. Background Technology
[0002] To improve the performance of Large Language Models (LLMs), a Mixture of Experts (MoE) network can be introduced into the LLM. Summary of the Invention
[0003] This disclosure provides a CPU, a decoding method, apparatus, device, medium, and product based on a hybrid expert network.
[0004] According to one aspect of this disclosure, a CPU is provided, comprising: a general-purpose computing function unit, configured to receive a decoding request and send the decoding request to an AI chip, the decoding request including pre-filled result information of historical tokens, so that the AI chip obtains an attention processing result of the historical tokens based on the pre-filled result information and determines a target token from the historical tokens; and an AI computing function unit, configured to receive the attention processing result of the target token, perform expert processing on the attention processing result to obtain an expert processing result of the target token, and send the expert processing result to the AI chip, so that the AI chip generates a current token based on the expert processing result.
[0005] According to another aspect of this disclosure, a decoding method based on a hybrid expert network is provided, applied to a CPU. The method includes: receiving a decoding request, the decoding request containing pre-filled result information of historical tokens; sending the decoding request to an AI chip, so that the AI chip obtains the attention processing result of the historical tokens based on the pre-filled result information, and determines a target token among the historical tokens; receiving the attention processing result of the target token, performing expert processing on the attention processing result to obtain an expert processing result of the target token; and sending the expert processing result to the AI chip, so that the AI chip generates a current token based on the expert processing result.
[0006] According to another aspect of this disclosure, a decoding method based on a hybrid expert network is provided, applied to an AI chip. The method includes: receiving a decoding request sent by a CPU, the decoding request containing pre-filled result information of historical tokens; obtaining attention processing results of the historical tokens based on the pre-filled result information, and determining a target token among the historical tokens; sending the attention processing result of the target token to the CPU, so that the CPU performs expert processing on the attention processing result to obtain an expert processing result of the target token; receiving the expert processing result, and generating a current token based on the expert processing result.
[0007] According to another aspect of this disclosure, a decoding apparatus based on a hybrid expert network is provided, applied to a CPU. The apparatus includes: a receiving module for receiving a decoding request, the decoding request containing pre-filled result information of historical tokens; a first sending module for sending the decoding request to an AI chip, so that the AI chip obtains the attention processing result of the historical tokens based on the pre-filled result information and determines a target token from the historical tokens; a processing module for receiving the attention processing result of the target token, performing expert processing on the attention processing result to obtain an expert processing result of the target token; and a second sending module for sending the expert processing result to the AI chip, so that the AI chip generates a current token based on the expert processing result.
[0008] According to another aspect of this disclosure, a decoding device based on a hybrid expert network is provided, applied to an AI chip. The device includes: a receiving module for receiving a decoding request sent by a CPU, the decoding request containing pre-filled result information of historical tokens; a processing module for obtaining attention processing results of the historical tokens based on the pre-filled result information, and determining a target token among the historical tokens; a sending module for sending the attention processing result of the target token to the CPU, so that the CPU performs expert processing on the attention processing result to obtain an expert processing result of the target token; and a generation module for receiving the expert processing result and generating a current token based on the expert processing result.
[0009] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.
[0010] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.
[0011] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.
[0012] According to embodiments of this disclosure, resource utilization can be balanced and overall throughput can be improved.
[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description
[0014] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:
[0015] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;
[0016] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;
[0017] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;
[0018] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;
[0019] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure;
[0020] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure;
[0021] Figure 7 This is a schematic diagram according to the seventh embodiment of the present disclosure;
[0022] Figure 8This is a schematic diagram of an electronic device used to implement the decoding method based on a hybrid expert network according to the embodiments of this disclosure. Detailed Implementation
[0023] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.
[0024] To better understand the embodiments of this disclosure, the relevant terms are explained as follows:
[0025] Central Processing Unit (CPU): The core computing and control unit of a computer system.
[0026] Artificial Intelligence (AI) Chips: Computing units for AI scenarios, specifically graphics processing units (GPUs), neural processing units (NPUs), tensor processing units (TPUs), etc.
[0027] The basic processing unit of LLM is called a token, which can be a text unit or an image unit, etc.
[0028] MoE Network: In the MoE architecture, each MoE network includes a first number (e.g., 256) of candidate expert networks. After the GPU receives the token, it uses a gating network to determine a second number (e.g., 8) of target expert networks from the first number of candidate expert networks, and then uses the second number of target expert networks to perform expert calculations on the token.
[0029] LLM is typically based on the Transformer architecture. The Transformer architecture model structure mainly includes an attention layer and a feed-forward network (FFN) layer. When the attention layer is computed, it needs to calculate the key-value (KV) data of the token, and then calculate the hidden state features of the token based on the KV data of the token. The FFN further processes the hidden state features of the token output by the attention layer to obtain the enhanced features, and then generates a new token through the subsequent output layer.
[0030] After introducing the MoE network, the FFN layer is specifically a MoE FFN, which includes multiple expert networks, each of which is an FFN.
[0031] To improve the inference performance of LLM, the inference stage can be divided into a prefill stage and a decoder stage. The prefill stage mainly calculates the key-value data of the token. After introducing the MoE network, the decode stage can be divided into a decode-attention stage and a decode-MoE FFN stage. The decode-attention stage is used for attention calculation, and the decode-MoE FFN stage is used for expert calculation.
[0032] The performance characteristics of each of the above stages are:
[0033] The Prefill stage has high computational power requirements but low bandwidth requirements; the Decode-Attention stage has high computational power requirements and high bandwidth requirements; the Decode-MoE FFN stage has low computational power requirements and low bandwidth requirements.
[0034] In related technologies, the processes of the Prefill stage, Decode-Attention stage, and Decode-MoE FFN stage are all executed by AI chips, such as GPUs, while the CPU is relatively idle. This can lead to problems such as uneven resource utilization and impact on overall throughput.
[0035] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure, which provides a CPU.
[0036] like Figure 1 As shown, the CPU 100 includes a general-purpose computing unit 101 and an AI computing unit 102. This CPU can be deployed within an AI server.
[0037] The general computing function unit 101 is used to receive a decoding request and send the decoding request to the AI chip. The decoding request contains pre-filled result information of historical tokens, so that the AI chip can obtain the attention processing result of the historical tokens based on the pre-filled result information and determine the target token in the historical tokens.
[0038] AI computing function unit 102 is configured to receive the attention processing result of the target token, perform expert processing on the attention processing result to obtain the expert processing result of the target token, and send the expert processing result to the AI chip so that the AI chip generates the current token based on the expert processing result.
[0039] Here, the current token refers to the token to be generated, and the historical token refers to the existing tokens associated with the current token.
[0040] For example, if a user sends the question text "How is the weather today?" to the LLM, the LLM will generate tokens one by one when replying. For example, it will first generate "Today", then generate "Sunny", and finally send all the generated tokens, such as "Today is sunny", back to the user as the reply text.
[0041] When generating "Today", existing input tokens, such as "Today", "Weather", and "How is it", can be used as historical tokens. When generating "Sunny", both existing input tokens and generated output tokens can be used as historical tokens, such as "Today", "Weather", "How is it", and the generated output token "Today".
[0042] In the PD (Prefill-Decode) separation scenario, the P (prefill) node can calculate the key-value data of the token. Specifically, the P node can be a GPU. The D (decode) node then generates the token based on the key-value data calculated by the P node.
[0043] The decoding stage includes an attention computation stage and an expert computation stage. In related technologies, both the attention computation stage and the expert computation stage are executed by AI chips, such as GPUs.
[0044] In this embodiment, considering the performance characteristics of the expert computation stage, the expert computation stage is offloaded to the CPU for execution, allowing the AI chip to focus on the attention stage.
[0045] Based on this, the CPU can be divided into a general-purpose computing unit and an AI computing unit. The AI computing unit is used to perform the expert calculations mentioned above, while the general-purpose computing unit is used to perform the general functions of the CPU.
[0046] To avoid interference between the two, the general computing functional unit and the AI computing functional unit can be independent of each other. For example, the general computing functional unit is the first chip, and the AI computing functional unit is the second chip. They are different chips and achieve independent functions.
[0047] The general-purpose computing function unit has input-output (IO) interfaces and communicates with the outside world through the IO interfaces.
[0048] For example, after P node calculates the pre-filled result, it can send a decoder request to the CPU. The CPU's general-purpose computing unit receives the decoder request through this I / O interface. The decoder request contains the pre-filled result information. Specifically, the pre-filled result can be the key-value data of the token, and the pre-filled result information can be the storage address of this key-value data.
[0049] After receiving the decoding request, the CPU's general computing unit sends it to the AI chip, which then performs attention calculations based on the decoding request.
[0050] Taking the GPU as an example of an AI chip, the GPU can obtain the KV data of the historical token corresponding to the current token from the P node according to the storage address of the KV data contained in the decoding request. It can then perform attention processing based on the KV data of the historical token to obtain the attention processing result of the historical token. Specifically, the attention processing result can be the hidden state features of the historical token.
[0051] In addition, AI chips can also identify target tokens from historical tokens.
[0052] The target token refers to the token deployed on that CPU by the target expert network.
[0053] For example, historical tokens include token1 and token2. Suppose that after calculation, the AI chip determines that the target expert network corresponding to token1 is EP1 and the target expert network corresponding to token2 is EP2. Suppose that the current target CPU is CPU1 and EP1 is deployed on CPU1, then the target token is token1.
[0054] The CPU's AI computing unit receives the attention processing result of the target token, performs expert processing on the attention processing result to obtain the expert processing result of the target token, and feeds it back to the AI chip. The AI chip generates the current token based on the expert processing result.
[0055] For example, the current CPU is CPU1. The AI computing function unit of CPU1 receives the attention processing result of token1, uses EP1 to perform expert calculation on the attention processing result, obtains the expert processing result of token1, and then feeds it back to the GPU mentioned above. The GPU performs subsequent processing on the expert processing results corresponding to each historical token to generate the current token.
[0056] The AI computing function unit can interact with the AI chip through the general computing function unit. For example, the attention processing results sent by the GPU are sent to the AI computing function unit through the general computing function unit, and the expert processing results obtained by the AI computing function unit are sent to the AI chip through the general computing function unit.
[0057] In this embodiment, the CPU includes an AI computing function unit that performs expert processing. This allows MoE computation to be offloaded from the AI chip to the CPU, thereby balancing the resource utilization of each hardware component and improving overall throughput.
[0058] Figure 2 This is a schematic diagram based on a second embodiment of the present disclosure, in which a CPU is provided.
[0059] like Figure 2 As shown, CPU 200 includes: a general computing function unit 201 and an AI computing function unit 202.
[0060] like Figure 2 As shown, the general-purpose computing functional unit 201 includes: a general-purpose computing logic unit, a general-purpose computing memory, an I / O control logic unit, and a general-purpose computing interconnect bus; the general-purpose computing logic unit, the general-purpose computing memory, and the I / O control logic unit are all connected to the general-purpose computing interconnect bus.
[0061] The IO control logic unit provides an IO interface for communication with external devices.
[0062] A general-purpose computing logic unit is used to execute general-purpose computing logic, such as scheduling decoding requests.
[0063] General-purpose computing memory is used to store data required for general computing logic, such as the scheduling rules for storing decoding requests. Specifically, it can be Double Data Rate Synchronous Dynamic Random Access Memory (DDR) or Low Power Double Data Rate Synchronous Dynamic Random Access Memory (LPDDR), and can be used in 8 or 16 channels.
[0064] The general-purpose computing interconnect bus is used to connect the I / O control logic unit, the general-purpose computing logic unit, and the general-purpose computing memory, enabling interconnection and communication between these units.
[0065] In this way, general computing functions can be implemented within the CPU based on the aforementioned units.
[0066] The AI computing functional unit 202 includes: an AI computing logic unit, an AI computing memory, and an AI computing interconnect bus; the AI computing logic unit and the AI computing memory are both connected to the AI computing interconnect bus, and the AI computing interconnect bus is connected to the general computing interconnect bus.
[0067] AI computing logic unit, used to perform expert computing functions.
[0068] AI computing memory is used to store the data required for expert calculations, such as storing the relevant parameters of the target expert network corresponding to the target token. Specifically, it can be a high-speed memory, such as high-bandwidth memory (HBM) or three-dimensional dynamic random access memory (3D-DRAM), which can use 8 or 16 channels.
[0069] The AI computing interconnect bus is used to connect the AI computing logic unit, AI computing memory and general computing interconnect bus to realize the interconnection and communication between the above units.
[0070] In this way, expert computing functions can be implemented within the CPU based on the aforementioned units.
[0071] Based on the above architecture:
[0072] Decoding requests are handled by the general computing function unit.
[0073] The decoding request is sent to the general computing logic unit via the IO control logic unit and the general computing interconnect bus.
[0074] Upon receiving a decoding request, the general-purpose computing logic unit can schedule it according to preset scheduling rules and allocate it to the corresponding AI chip.
[0075] The scheduling rule can be a data parallelism (DP) method. For example, if the decoding request includes a first decoding request and a second decoding request, and the AI chip includes a first AI chip and a second AI chip, then the CPU can assign the first decoding request to the first AI chip for processing and the second decoding request to the second AI chip for processing.
[0076] Furthermore, the aforementioned scheduling rules can be recorded in general-purpose computing memory. Thus, the general-purpose computing logic unit can obtain the scheduling rules from the general-purpose computing memory, schedule multiple received decoding requests based on the scheduling rules, and send the decoding requests to the corresponding AI chip.
[0077] When sending a decoding request, taking the first decoding request as an example, the general computing logic unit sends the first decoding request to the first AI chip via the general computing interconnect bus and the IO control logic unit.
[0078] In this way, based on general-purpose computing logic units and general-purpose computing memory, general-purpose computing functions can be completed independently, avoiding interference with AI computing functions and improving overall performance.
[0079] Expert processing is handled by the AI computing functional unit.
[0080] The attention processing result is sent from the AI chip to the AI computing logic unit via the IO control logic unit, the general computing interconnect bus, and the AI computing interconnect bus.
[0081] After receiving the attention processing result of the target token, the AI computing logic unit uses pre-deployed target experts to perform expert calculations on the attention processing result of the target token, and obtains the expert processing result of the target token.
[0082] Furthermore, the target expert network corresponding to the target token can be deployed in the AI computing memory. That is, the AI computing memory stores the weights and other parameters of the target expert network. In this way, the AI computing logic unit can obtain the target expert network from the AI computing memory, use the target expert network to perform expert calculations on the attention processing results of the received target token, and obtain the expert processing result.
[0083] Assuming the attention processing result mentioned above comes from the first AI chip, the AI computing logic unit calculates the expert processing result of the target token and then sends it to the first AI chip via the AI computing interconnect bus, the general computing interconnect bus and the IO control logic unit.
[0084] In this way, based on the AI computing logic unit and AI computing memory, expert computing functions can be completed independently, avoiding interference with general computing functions and improving overall performance.
[0085] Figure 3 This is a schematic diagram according to a third embodiment of the present disclosure. This embodiment illustrates the interconnection between a CPU and an AI chip.
[0086] like Figure 3 As shown, in an LLM scenario, there can be multiple CPUs and multiple AI chips.
[0087] CPU 300 includes: a general computing function unit 301 and an AI computing function unit 302.
[0088] The general-purpose computing function unit 301 acts as a control plane, for example, scheduling decoding requests and sending them to the AI chip. Specifically, the decoding requests can be scheduled using a dynamic programming (DP) approach.
[0089] The AI chip is used to perform attention calculations. That is, it performs attention calculations on historical tokens to obtain the attention processing result. Different AI chips can process this in parallel using dynamic programming (DP). For example, if the first AI chip receives a first decoding request and the second AI chip receives a second decoding request, then the first AI chip can perform attention calculations based on the first decoding request, and the second AI chip can perform attention calculations based on the second decoding request. Furthermore, the first and second AI chips execute the attention calculation process in parallel.
[0090] Furthermore, for each AI chip, the historical tokens corresponding to the received decoding requests can be determined, and the target CPUs corresponding to each historical token can be identified and assigned to the corresponding CPUs for expert computation. For example, the historical tokens corresponding to the first AI chip include token1 and token2. The target expert network corresponding to token1 is EP1, and the target expert network corresponding to token2 is EP2. EP1 is deployed on CPU1, and EP2 is deployed on CPU2. Then, the first AI chip sends token1 to CPU1, and CPU1 uses EP1 to perform expert computation on token1. The first AI chip sends token2 to CPU2, and CPU2 uses EP2 to perform expert computation on token2.
[0091] AI computing unit 302 is used to perform expert computations. Expert computations can be performed using expert parallelism (EP), for example, CPU1 and CPU2 mentioned above can perform expert computations in parallel. Furthermore, each CPU can deploy one or more expert networks, and multiple expert networks on each CPU can be executed in parallel. For example, if the target expert network corresponding to token1 also includes EP3, and EP3 is deployed on CPU1, then CPU1 can use EP1 and EP3 in parallel to perform expert computations on token1.
[0092] The CPU and the AI chip communicate with each other through an interconnection system. The CPU and the AI chip can be interconnected via an XLink link, a Peripheral Component Interconnect Express (PCIe) link, or a network card link.
[0093] Specifically, the CPU and AI chip are connected to the XLink switch; or, the CPU and AI chip are interconnected via PCIe or a PCIe switch, and the AI chip is interconnected via an XLink switch; or, the CPU and AI chip are interconnected via PCIe or a PCIe switch, and the AI chip is interconnected via a network card.
[0094] Among them, the XLink link is a high-speed interconnect link between devices (such as between GPUs, or between GPU and CPU), which can include NVLink link, UALINK (Ultra Accelerator Link) link, UB (UnifiedBus) link, etc.
[0095] Network interface cards (NICs) can specifically include: Remote Direct Memory Access (RDMA) NICs, Infiniband (IB) NICs, or RDMA over Converged Ethernet (RoCE) NICs, etc.
[0096] In this way, the CPU and AI chip can be interconnected in multiple ways, improving flexibility.
[0097] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure. This embodiment provides a decoding method based on a hybrid expert network, applied to a CPU. The method includes:
[0098] 401. Receive a decoding request, wherein the decoding request contains pre-filled result information of the historical token.
[0099] 402. Send the decoding request to the AI chip so that the AI chip can obtain the attention processing result of the historical token based on the pre-filled result information and determine the target token in the historical token.
[0100] 403. Receive the attention processing result of the target token, and perform expert processing on the attention processing result to obtain the expert processing result of the target token.
[0101] 404. Send the expert processing result to the AI chip so that the AI chip generates the current token based on the expert processing result.
[0102] In the PD (Prefill-Decode) separation scenario, the P (prefill) node can calculate the key-value data of the token. Specifically, the P node can be a GPU. The D (decode) node then generates the token based on the key-value data calculated by the P node.
[0103] The decoding stage includes an attention computation stage and an expert computation stage. In related technologies, both the attention computation stage and the expert computation stage are executed by AI chips, such as GPUs.
[0104] In this embodiment, considering the performance characteristics of the expert computation stage, the expert computation stage is offloaded to the CPU for execution, allowing the AI chip to focus on the attention stage.
[0105] Based on this, the CPU can be divided into a general-purpose computing unit and an AI computing unit. The AI computing unit is used to perform the expert calculations mentioned above, while the general-purpose computing unit is used to perform the general functions of the CPU.
[0106] To avoid interference between the two, the general computing functional unit and the AI computing functional unit can be independent of each other. For example, the general computing functional unit is the first chip, and the AI computing functional unit is the second chip. They are different chips and achieve independent functions.
[0107] The general-purpose computing function unit has input-output (IO) interfaces and communicates with the outside world through the IO interfaces.
[0108] For example, after P node calculates the pre-filled result, it can send a decoder request to the CPU. The CPU's general-purpose computing unit receives the decoder request through this I / O interface. The decoder request contains the pre-filled result information. Specifically, the pre-filled result can be the key-value data of the token, and the pre-filled result information can be the storage address of this key-value data.
[0109] After receiving the decoding request, the CPU's general computing unit sends it to the AI chip, which then performs attention calculations based on the decoding request.
[0110] Taking the GPU as an example of an AI chip, the GPU can obtain the KV data of the historical token corresponding to the current token from the P node according to the storage address of the KV data contained in the decoding request. It can then perform attention processing based on the KV data of the historical token to obtain the attention processing result of the historical token. Specifically, the attention processing result can be the hidden state features of the historical token.
[0111] In addition, AI chips can also identify target tokens from historical tokens.
[0112] The target token refers to the token deployed on that CPU by the target expert network.
[0113] For example, historical tokens include token1 and token2. Suppose that after calculation, the AI chip determines that the target expert network corresponding to token1 is EP1 and the target expert network corresponding to token2 is EP2. Suppose that the current target CPU is CPU1 and EP1 is deployed on CPU1, then the target token is token1.
[0114] The CPU's AI computing unit receives the attention processing result of the target token, performs expert processing on the attention processing result to obtain the expert processing result of the target token, and feeds it back to the AI chip. The AI chip generates the current token based on the expert processing result.
[0115] For example, the current CPU is CPU1. The AI computing function unit of CPU1 receives the attention processing result of token1, uses EP1 to perform expert calculation on the attention processing result, obtains the expert processing result of token1, and then feeds it back to the GPU mentioned above. The GPU performs subsequent processing on the expert processing results corresponding to each historical token to generate the current token.
[0116] The AI computing function unit can interact with the AI chip through the general computing function unit. For example, the attention processing results sent by the GPU are sent to the AI computing function unit through the general computing function unit, and the expert processing results obtained by the AI computing function unit are sent to the AI chip through the general computing function unit.
[0117] In this embodiment, the CPU includes an AI computing function unit that performs expert processing. This allows MoE computation to be offloaded from the AI chip to the CPU, thereby balancing the resource utilization of each hardware component and improving overall throughput.
[0118] In some embodiments, the decoding request is processed by a general-purpose computing unit within the CPU; the expert processing is performed by an AI computing unit within the CPU; and the general-purpose computing unit and the AI computing unit are independent of each other.
[0119] Decoding requests are handled by the general computing function unit.
[0120] The decoding request is sent to the general computing logic unit via the IO control logic unit and the general computing interconnect bus.
[0121] Upon receiving a decoding request, the general-purpose computing logic unit can schedule it according to preset scheduling rules and allocate it to the corresponding AI chip.
[0122] The scheduling rule can be a data parallelism (DP) method. For example, if the decoding request includes a first decoding request and a second decoding request, and the AI chip includes a first AI chip and a second AI chip, then the CPU can assign the first decoding request to the first AI chip for processing and the second decoding request to the second AI chip for processing.
[0123] Furthermore, the aforementioned scheduling rules can be recorded in general-purpose computing memory. Thus, the general-purpose computing logic unit can obtain the scheduling rules from the general-purpose computing memory, schedule multiple received decoding requests based on the scheduling rules, and send the decoding requests to the corresponding AI chip.
[0124] When sending a decoding request, taking the first decoding request as an example, the general computing logic unit sends the first decoding request to the first AI chip via the general computing interconnect bus and the IO control logic unit.
[0125] In this way, based on general-purpose computing logic units and general-purpose computing memory, general-purpose computing functions can be completed independently, avoiding interference with AI computing functions and improving overall performance.
[0126] In some embodiments, receiving the attention processing result of the target token and performing expert processing on the attention processing result to obtain the expert processing result of the target token includes:
[0127] Receive the attention processing result of the target token;
[0128] The attention processing result is processed by an expert network pre-deployed within the CPU corresponding to the target token to obtain the expert processing result of the target token.
[0129] Expert processing is handled by the AI computing functional unit.
[0130] The attention processing result is sent from the AI chip to the AI computing logic unit via the IO control logic unit, the general computing interconnect bus, and the AI computing interconnect bus.
[0131] After receiving the attention processing result of the target token, the AI computing logic unit uses pre-deployed target experts to perform expert calculations on the attention processing result of the target token, and obtains the expert processing result of the target token.
[0132] Furthermore, the target expert network corresponding to the target token can be deployed in the AI computing memory. That is, the AI computing memory stores the weights and other parameters of the target expert network. In this way, the AI computing logic unit can obtain the target expert network from the AI computing memory, use the target expert network to perform expert calculations on the attention processing results of the received target token, and obtain the expert processing result.
[0133] Assuming the attention processing result mentioned above comes from the first AI chip, the AI computing logic unit calculates the expert processing result of the target token and then sends it to the first AI chip via the AI computing interconnect bus, the general computing interconnect bus and the IO control logic unit.
[0134] In this way, based on the AI computing logic unit and AI computing memory, expert computing functions can be completed independently, avoiding interference with general computing functions and improving overall performance.
[0135] Figure 5 This is a schematic diagram according to the fifth embodiment of the present disclosure. This embodiment provides a decoding method based on a hybrid expert network, applied to an AI chip. The method includes:
[0136] 501. Receive a decoding request sent by the CPU, wherein the decoding request contains pre-filled result information of the historical token.
[0137] 502. Obtain the attention processing result of the historical token based on the pre-filled result information, and determine the target token in the historical token.
[0138] 503. Send the attention processing result of the target token to the CPU so that the CPU can perform expert processing on the attention processing result to obtain the expert processing result of the target token.
[0139] 504. Receive the expert processing result and generate the current token based on the expert processing result.
[0140] For example, after P node calculates the pre-filled result, it can send a decoder request to the CPU. The CPU's general-purpose computing unit receives the decoder request through this I / O interface. The decoder request contains the pre-filled result information. Specifically, the pre-filled result can be the key-value data of the token, and the pre-filled result information can be the storage address of this key-value data.
[0141] After receiving the decoding request, the CPU's general computing unit sends it to the AI chip, which then performs attention calculations based on the decoding request.
[0142] Taking the GPU as an example of an AI chip, the GPU can obtain the KV data of the historical token corresponding to the current token from the P node according to the storage address of the KV data contained in the decoding request. It can then perform attention processing based on the KV data of the historical token to obtain the attention processing result of the historical token. Specifically, the attention processing result can be the hidden state features of the historical token.
[0143] In addition, AI chips can also identify target tokens from historical tokens.
[0144] The target token refers to the token deployed on that CPU by the target expert network.
[0145] For example, historical tokens include token1 and token2. Suppose that after calculation, the AI chip determines that the target expert network corresponding to token1 is EP1 and the target expert network corresponding to token2 is EP2. Suppose that the current target CPU is CPU1 and EP1 is deployed on CPU1, then the target token is token1.
[0146] The CPU's AI computing unit receives the attention processing result of the target token, performs expert processing on the attention processing result to obtain the expert processing result of the target token, and feeds it back to the AI chip. The AI chip generates the current token based on the expert processing result.
[0147] For example, the current CPU is CPU1. The AI computing function unit of CPU1 receives the attention processing result of token1, uses EP1 to perform expert calculation on the attention processing result, obtains the expert processing result of token1, and then feeds it back to the GPU mentioned above. The GPU performs subsequent processing on the expert processing results corresponding to each historical token to generate the current token.
[0148] In this embodiment, the CPU includes an AI computing function unit that performs expert processing. This allows MoE computation to be offloaded from the AI chip to the CPU, thereby balancing the resource utilization of each hardware component and improving overall throughput.
[0149] Figure 6 This is a schematic diagram according to the sixth embodiment of the present disclosure. This embodiment provides a decoding device based on a hybrid expert network. The device is applied to a CPU, such as... Figure 6 As shown, the device 600 includes: a receiving module 601, a first transmitting module 602, a processing module 603, and a second transmitting module 604.
[0150] The receiving module 601 is used to receive a decoding request, wherein the decoding request contains pre-filled result information of the historical token;
[0151] The first sending module 602 is used to send the decoding request to the AI chip, so that the AI chip can obtain the attention processing result of the historical token based on the pre-filled result information, and determine the target token in the historical token;
[0152] The processing module 603 is used to receive the attention processing result of the target token, perform expert processing on the attention processing result, and obtain the expert processing result of the target token.
[0153] The second sending module 604 is used to send the expert processing result to the AI chip so that the AI chip generates the current token based on the expert processing result.
[0154] In this embodiment, the CPU includes an AI computing function unit that performs expert processing. This allows MoE computation to be offloaded from the AI chip to the CPU, thereby balancing the resource utilization of each hardware component and improving overall throughput.
[0155] In some embodiments, the decoding request is processed by a general-purpose computing unit within the CPU; the expert processing is performed by an AI computing unit within the CPU; and the general-purpose computing unit and the AI computing unit are independent of each other.
[0156] In this way, based on general-purpose computing logic units and general-purpose computing memory, general-purpose computing functions can be completed independently, avoiding interference with AI computing functions and improving overall performance.
[0157] In some embodiments, the processing module 603 is further configured to:
[0158] Receive the attention processing result of the target token;
[0159] The attention processing result is processed by an expert network pre-deployed within the CPU corresponding to the target token to obtain the expert processing result of the target token.
[0160] In this way, based on the AI computing logic unit and AI computing memory, expert computing functions can be completed independently, avoiding interference with general computing functions and improving overall performance.
[0161] Figure 7 This is a schematic diagram based on the seventh embodiment of the present disclosure. This embodiment provides a decoding device based on a hybrid expert network. This device is applied to AI chips, such as... Figure 7As shown, the device 700 includes: a receiving module 701, a processing module 702, a sending module 703, and a generating module 704.
[0162] The receiving module 701 is used to receive a decoding request sent by the CPU, wherein the decoding request contains pre-filled result information of the historical token;
[0163] Processing module 702 is used to obtain the attention processing result of the historical token based on the pre-filled result information, and determine the target token in the historical token;
[0164] The sending module 703 is used to send the attention processing result of the target token to the CPU, so that the CPU performs expert processing on the attention processing result to obtain the expert processing result of the target token;
[0165] The generation module 704 is used to receive the expert processing result and generate the current token based on the expert processing result.
[0166] In this embodiment, the CPU includes an AI computing function unit that performs expert processing. This allows MoE computation to be offloaded from the AI chip to the CPU, thereby balancing the resource utilization of each hardware component and improving overall throughput.
[0167] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.
[0168] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.
[0169] It is understandable that, unless otherwise specified, the order of steps in the process indicates that the temporal relationship between these steps is not limited.
[0170] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.
[0171] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.
[0172] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.
[0173] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0174] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0175] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the decoding method based on hybrid expert networks. For example, in some embodiments, the decoding method based on hybrid expert networks can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the decoding method based on hybrid expert networks described above can be performed. Alternatively, in other embodiments, computing unit 801 may be configured to perform a decoding method based on a hybrid expert network by any other suitable means (e.g., by means of firmware).
[0176] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0177] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable task processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0178] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.
[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0180] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.
[0181] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service system that addresses the shortcomings of traditional physical hosts and VPS (Virtual Private Server) services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.
[0182] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.
[0183] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.
Claims
1. A CPU, comprising: A general computing function unit is used to receive a decoding request and send the decoding request to an AI chip. The decoding request contains pre-filled result information of historical tokens, so that the AI chip can obtain the attention processing result of the historical tokens based on the pre-filled result information and determine the target token in the historical tokens. The AI computing function unit is configured to receive the attention processing result of the target token, perform expert processing on the attention processing result to obtain the expert processing result of the target token, and send the expert processing result to the AI chip so that the AI chip generates the current token based on the expert processing result.
2. The CPU according to claim 1, wherein, The general computing functional unit includes: General-purpose computing logic unit, general-purpose computing memory, I / O control logic unit, and general-purpose computing interconnect bus; The general-purpose computing logic unit, general-purpose computing memory, and I / O control logic unit are all connected to the general-purpose computing interconnect bus.
3. The CPU according to claim 2, wherein, The AI computing functional unit includes: AI computing logic unit, AI computing memory and AI computing interconnect bus; The AI computing logic unit and the AI computing memory are both connected to the AI computing interconnect bus, and the AI computing interconnect bus is connected to the general computing interconnect bus.
4. The CPU according to claim 3, wherein, The attention processing result is sent from the AI chip to the AI computing logic unit via the IO control logic unit, the general computing interconnect bus, and the AI computing interconnect bus; The AI computing memory pre-deploys a target expert network corresponding to the target token. The AI computing logic unit is specifically used to: perform expert processing on the attention processing result of the target token using the target expert network, so as to obtain the expert processing result.
5. The CPU according to claim 2, wherein, The decoding request is sent to the general computing logic unit via the IO control logic unit and the general computing interconnect bus; The general-purpose computing memory contains pre-recorded scheduling rules; The general computing logic unit is specifically used to: send the decoding request to the AI chip according to the scheduling rules.
6. The CPU according to claim 1, wherein, The CPU and the AI chip are interconnected via an XLink link, a PCIe link, or a network card link.
7. A decoding method based on a hybrid expert network, applied to a CPU, the method comprising: Receive a decoding request, which contains pre-filled result information of historical tokens; The decoding request is sent to the AI chip so that the AI chip can obtain the attention processing result of the historical token based on the pre-filled result information and determine the target token in the historical token; Receive the attention processing result of the target token, and perform expert processing on the attention processing result to obtain the expert processing result of the target token; The expert processing result is sent to the AI chip so that the AI chip can generate the current token based on the expert processing result.
8. The method according to claim 7, wherein, The decoding request is processed by the general-purpose computing unit within the CPU; The expert processing is performed using the AI computing function unit within the CPU; The general computing logic unit and the AI computing logic unit are independent of each other.
9. The method according to claim 7, wherein, The process of receiving the attention processing result of the target token and performing expert processing on the attention processing result to obtain the expert processing result of the target token includes: Receive the attention processing result of the target token; The attention processing result is processed by an expert network pre-deployed within the CPU corresponding to the target token to obtain the expert processing result of the target token.
10. A decoding method based on a hybrid expert network, applied to an AI chip, the method comprising: Receive a decoding request sent by the CPU, the decoding request containing pre-filled result information of historical tokens; Based on the pre-filled result information, the attention processing result of the historical token is obtained, and the target token is determined in the historical token; The attention processing result of the target token is sent to the CPU so that the CPU can perform expert processing on the attention processing result to obtain the expert processing result of the target token; Receive the expert processing result and generate the current token based on the expert processing result.
11. A decoding device based on a hybrid expert network, applied to a CPU, the device comprising: A receiving module is used to receive a decoding request, wherein the decoding request contains pre-filled result information of the historical token; The first sending module is used to send the decoding request to the AI chip, so that the AI chip can obtain the attention processing result of the historical token based on the pre-filled result information, and determine the target token in the historical token; The processing module is used to receive the attention processing result of the target token, perform expert processing on the attention processing result, and obtain the expert processing result of the target token. The second sending module is used to send the expert processing result to the AI chip, so that the AI chip generates the current token based on the expert processing result.
12. A decoding device based on a hybrid expert network, applied to an AI chip, the device comprising: The receiving module is used to receive decoding requests sent by the CPU, wherein the decoding requests contain pre-filled result information of historical tokens; The processing module is used to obtain the attention processing result of the historical token based on the pre-filled result information, and to determine the target token in the historical token; The sending module is used to send the attention processing result of the target token to the CPU, so that the CPU can perform expert processing on the attention processing result to obtain the expert processing result of the target token; The generation module is used to receive the expert processing result and generate the current token based on the expert processing result.
13. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-6.
14. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-6.
15. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-6.