Processing chips, processing systems, and electronic devices

Abstract

This invention discloses a processing chip, a processing system, and an electronic device. In this invention, each processing unit within the processing chip is assigned to either a pre-filling instance or a decoding instance. Processing units assigned to pre-filling instances handle pre-filling tasks, while those assigned to decoding instances handle decoding tasks. A weight register is set in each processing unit, and a bandwidth allocation circuit unit is included in the processing chip. This bandwidth allocation circuit unit controls memory access for each processing unit based on its bandwidth weight, ensuring that the bandwidth usage of memory by each processing unit matches its bandwidth weight. Therefore, this invention can separately configure computing resources and bandwidth resources for pre-filling and decoding instances, thereby supporting the separation of pre-filling and decoding instances within a single chip.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically to a processing chip, a processing system, and an electronic device. Background Technology

[0002] Currently, large language model inference can be divided into two stages: the prefill stage and the decoding stage. In the prefill stage, the large language model needs to process all input at once to build a key-value cache and the first token. In the decoding stage, the large language model needs to generate subsequent tokens one by one based on the previously generated key-value cache and the first token. Due to different computational requirements, the prefill and decoding stages of a large language model can be classified as computationally intensive (i.e., execution speed mainly depends on the processor's computing power) and memory-intensive (i.e., execution speed mainly depends on memory read / write speed), respectively—two completely different computational modes. In existing technologies, to meet the computational needs of their respective stages, the prefill and decoding stages of a large language model are usually deployed on different hardware devices. However, while this approach optimizes resource utilization and latency, it also introduces architectural complexity and additional performance overhead (e.g., key-value cache transfer and storage redundancy). Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a processing chip, a processing system, and an electronic device to support the separation of pre-filled instances and decoding instances in a single chip.

[0004] In a first aspect, embodiments of the present invention aim to provide a processing chip, the processing chip comprising: Multiple processing units are provided, wherein each processing unit is assigned to a pre-fill instance or a decoding instance. The processing unit assigned to the pre-fill instance is used to process the pre-fill task, and the processing unit assigned to the decoding instance is used to process the decoding task. Each processing unit includes a weight register for storing the bandwidth weight of the processing unit. Multiple storage units are provided for access by the multiple processing units; A bandwidth allocation circuit unit is disposed between each of the processing units and each of the storage units. The bandwidth allocation circuit unit is configured to perform memory access control on each of the processing units according to the bandwidth weights, so that the bandwidth usage of each processing unit on the storage units matches the bandwidth weight of each processing unit; and... A packet-on-chip network is used to realize communication connections between each of the processing units and the bandwidth allocation circuit unit, and to realize communication connections between the bandwidth allocation circuit unit and each of the storage units.

[0005] In some embodiments, the number of processing units assigned to the pre-filled instance is greater than the number of processing units assigned to the decoding instance, and the bandwidth utilization rate of the processing units assigned to the pre-filled instance to the plurality of storage units is less than the bandwidth utilization rate of the processing units assigned to the decoding instance to the plurality of storage units.

[0006] In some embodiments, the bandwidth allocation circuit unit is specifically configured as follows: Initialize the token bucket of each of the processing units; Receive memory access requests sent by each of the processing units; Each memory access request is sent to the corresponding storage unit according to the token bucket and the bandwidth weight, so as to perform memory access control on each processing unit.

[0007] In some embodiments, the bandwidth allocation circuit unit is specifically configured as follows: The token bucket of each processing unit is replenished with tokens according to the bandwidth weight; High-priority processing units and low-priority processing unit groups are determined based on the number of tokens in each token bucket. The high-priority processing unit is the processing unit that contains the most tokens in the token bucket, and the low-priority processing unit group includes other processing units besides the high-priority processing unit. The memory access requests of the high-priority processing unit are sent to the corresponding storage unit in a manner that utilizes the maximum bandwidth, and the memory access requests of each processing unit in the low-priority processing unit group are sent to the corresponding storage unit in a manner that utilizes the remaining bandwidth. Tokens are deducted from the token bucket of the high-priority processing unit according to the bandwidth weight.

[0008] In some embodiments, the number of tokens replenished to the token bucket of each processing unit is equal to the bandwidth weight value of each processing unit, and the number of tokens deducted from the token bucket of the high-priority processing unit is equal to the sum of the bandwidth weight values ​​of each processing unit.

[0009] In some embodiments, the processing chip further includes: The first logic module is configured to receive bandwidth resource configuration information, determine the bandwidth weight of each processing unit according to the bandwidth resource configuration information, and store each bandwidth weight into the corresponding bandwidth register.

[0010] In some embodiments, the first logic module is specifically configured as follows: The pre-fill bandwidth weight of the pre-filled instance and the decoding bandwidth weight of the decoding instance are determined based on the bandwidth resource configuration information. For the pre-filled instance, the pre-filled bandwidth weight is evenly distributed according to the number of processing units assigned to the pre-filled instance to determine the bandwidth weight of each processing unit in the pre-filled instance. For the decoding instance, the decoding bandwidth weight is evenly distributed according to the number of processing units assigned to the decoding instance to determine the bandwidth weight of each processing unit in the decoding instance.

[0011] In some embodiments, the processing chip further includes: The second logic module is configured to receive computing power resource configuration information, determine the computing power resource allocation ratio according to the computing power resource configuration information, and allocate each of the processing units to the pre-filled instance or the decoding instance according to the computing power resource allocation ratio.

[0012] In some embodiments, the packet on-chip network includes: A first on-chip network is disposed between the plurality of processing units and the bandwidth allocation circuit unit, and the first on-chip network is used to realize the communication connection between each of the processing units and the bandwidth allocation circuit unit; The second on-chip network is disposed between the bandwidth allocation circuit unit and the plurality of storage units, and the first on-chip network is used to realize the communication connection between the bandwidth allocation circuit unit and each of the storage units.

[0013] In some embodiments, the processing chip is a graphics processor, the processing unit is a streaming multiprocessor, and the storage unit is a last-level cache.

[0014] In a second aspect, embodiments of the present invention aim to provide a processing system, the processing system including the processing chip as described in the first aspect.

[0015] Thirdly, embodiments of the present invention aim to provide an electronic device, the electronic device including the processing system as described in the second aspect.

[0016] This invention assigns each processing unit in a processing chip to either a pre-filling instance or a decoding instance. Processing units assigned to pre-filling instances handle pre-filling tasks, while those assigned to decoding instances handle decoding tasks. A weight register is set in each processing unit, and a bandwidth allocation circuit unit is included in the processing chip. This bandwidth allocation circuit unit controls memory access for each processing unit based on its bandwidth weight, ensuring that the bandwidth usage of each processing unit matches its bandwidth weight. Therefore, this invention can separately configure computing and bandwidth resources for pre-filling and decoding instances, thus supporting the separation of pre-filling and decoding instances within a single chip. Attached Figure Description

[0017] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which: Figure 1 This is a schematic diagram of the processing chip according to an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the instance division of an embodiment of the present invention; Figure 3 This is a flowchart of the memory access control method according to an embodiment of the present invention; Figure 4 This is a flowchart of the memory access control method according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the memory access control process according to an embodiment of the present invention; Figure 6 This is a flowchart of a computing resource allocation method according to an embodiment of the present invention; Figure 7 This is a flowchart of a bandwidth resource configuration method according to an embodiment of the present invention; Figure 8 This is a flowchart of the bandwidth weight determination method according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the processing system according to an embodiment of the present invention; Figure 10 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0018] The present application is described below based on embodiments, but it is not limited to these embodiments. In the detailed description of the present application below, certain specific details are described in detail. Those skilled in the art can fully understand the present application without these details. To avoid obscuring the substance of the present application, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0019] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0020] Unless the context explicitly requires it, words such as "including" or "contains" throughout the application should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0021] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0022] The solutions described in this specification and embodiments, if involving the processing of personal information, will be processed only on the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0023] Figure 1 This is a schematic diagram of the processing chip according to an embodiment of the present invention. It is intended to illustrate that... Figure 1 The processing chip shown can be understood as an integrated circuit specifically designed to perform computational, logic control, and data processing tasks. This processing chip can be used to deploy large language models. Furthermore, this processing chip can receive and process task requests from different user objects in batches based on the deployed large language model. Figure 1 As shown, the processing chip 1 includes multiple processing units 11, multiple storage units 12, and a bandwidth allocation circuit unit 13.

[0024] The processing unit 11 can be a basic hardware module within the processing chip 1 responsible for performing calculations, control, and data processing. In this embodiment of the invention, each processing unit 11 can be assigned to a pre-filling instance or a decoding instance. The processing unit 11 assigned to the pre-filling instance can be used to process pre-filling tasks, and the processing unit 11 assigned to the decoding instance can be used to process decoding tasks. It is worth noting that the pre-filling instance and the decoding instance can be considered as a group of processing units that are separated from each other in terms of scheduling, batch processing, and computing hardware resources within the same processing chip 1, respectively (this separation can be implemented at the physical level or at the logical level, and this application does not impose any limitations on this), and can run simultaneously. Specifically, having the processing unit 11 assigned to the pre-filling instance process pre-filling tasks means assigning the computational tasks of the pre-filling stage to the processing unit 11 assigned to the pre-filling instance for execution. Similarly, having the processing unit 11 assigned to the decoding instance process decoding tasks means assigning the computational tasks of the decoding stage to the processing unit 11 assigned to the decoding instance for execution. Furthermore, in this embodiment of the invention, each processing unit 11 may include a weight register (not shown in the figure). The weight register may be used to store the bandwidth weights of the processing unit 11 (as shown by ω1, ω2, ω3, and ω4). The bandwidth weights may be used to characterize the bandwidth requirements of each processing unit 11 for the storage units, and may determine the bandwidth occupancy ratio of each processing unit 11 for the multiple storage units 12. Optionally, depending on the setting of the bandwidth weights, the weight register may be implemented by any register of the processing unit 11 with a matching number of bits (e.g., 8 bits), and this application does not impose any limitation on this.

[0025] Schematic, processing chip 1 may be a graphics processing unit (GPU) chip. Processing unit 11 may be a graphics processing cluster (GPC) or a streaming multiprocessor (SM). It should be understood that in some embodiments, the processing chip may also be a central processing unit (CPU) chip, a tensor processing unit (TPU) chip, an embedded neural network processing unit (NPU) chip, a digital signal processing (DSP) chip, a language processing unit (LPU) chip, or any other type of inference chip, etc., and this application does not impose any limitations on this. Furthermore, when the processing chip is an embedded neural network processor, the processing unit may be a processing engine (PE). When the processing chip is an embedded neural network processor or a tensor processor chip, the processing unit may be a processing engine (PE) or a matrix multiply unit (MXU). When the processing chip is a language processor chip, the processing unit can be a TSP (Tensor Streaming Processor Slice).

[0026] Storage unit 12 can be a basic hardware module inside processing chip 1 responsible for data storage and access. Storage unit 12 can be used to store data required for the operation of each processing unit 11 or to store relevant data generated by each processing unit 11 during operation. In this embodiment of the invention, each storage unit 12 can be accessed by each processing unit 11 to enable the processing unit 11 to store and read task request data, generated key-value caches, the first word, and subsequent words, etc. Optionally, storage unit 12 can be memory, various levels of cache (e.g., the last level cache), or external storage, etc., and this application does not impose any limitations on this.

[0027] A bandwidth allocation circuit unit 13 can be disposed between each processing unit 11 and each memory unit 12. The bandwidth allocation circuit unit 13 can be a hardware circuit module for implementing memory access control. In this embodiment of the invention, the bandwidth allocation circuit unit 13 can be configured to perform memory access control on each processing unit 11 according to bandwidth weights, so that the bandwidth occupancy of each processing unit 11 on the memory unit 12 matches the bandwidth weight of each processing unit 11. Optionally, the bandwidth allocation circuit unit 13 can be obtained by having its operating logic described by a relevant person using a hardware description language (such as VHDL or Verilog), and then mapping the operating logic onto a programmable integrated circuit (e.g., a Field Programmable Gate Array, FPGA) by programming.

[0028] Therefore, by dividing each processing unit in the processing chip into a pre-fill instance or a decoding instance, the processing units assigned to the pre-fill instance handle the pre-fill task, and the processing units assigned to the decoding instance handle the decoding task. In addition, a weight register is set in each processing unit, and a bandwidth allocation circuit unit is set in the processing chip. The bandwidth allocation circuit unit controls the memory access of each processing unit according to the bandwidth weight, so that the bandwidth occupation of the memory unit by each processing unit matches the bandwidth weight of each processing unit. The embodiments of the present invention can realize the configuration of computing power resources and bandwidth resources for pre-fill instances and decoding instances respectively, thereby supporting the separation of pre-fill instances and decoding instances in a single chip.

[0029] Furthermore, in addition to the multiple processing units 11, multiple storage units 12, and bandwidth allocation circuit unit 13, the processing chip 1 may also include a packet on-chip network. The packet on-chip network can be used to respectively implement communication connections between each processing unit 11 and the bandwidth allocation circuit unit 13, and to implement communication connections between the bandwidth allocation circuit unit 13 and each storage unit 12. Schematably, as one configuration, the packet on-chip network can be configured to include a first on-chip network (as shown by the connection between the processing unit 11 and the bandwidth allocation circuit unit 13) and a second on-chip network (as shown by the connection between the bandwidth allocation circuit unit 13 and the storage unit 12). The first on-chip network can be disposed between the multiple processing units 11 and the bandwidth allocation circuit unit 13 to implement communication connections between each processing unit 11 and the bandwidth allocation circuit unit 13. The second on-chip network can be disposed between the bandwidth allocation circuit unit 13 and the multiple storage units 12 to implement communication connections between the bandwidth allocation circuit unit 13 and each storage unit 12. Optionally, the on-chip network can be implemented by setting up a bus and routing module. This application does not limit the specific implementation method of the on-chip network.

[0030] Optionally, to simultaneously meet the high computing power requirements of the pre-filled instances and the high memory access requirements of the decoding instances, in this embodiment of the invention, the number of processing units allocated to the pre-filled instances can be set to be greater than the number of processing units allocated to the decoding instances. The bandwidth utilization rate of the processing units allocated to the pre-filled instances across multiple memory units can be set to be less than the bandwidth utilization rate of the processing units allocated to the decoding instances across multiple memory units. The sum of the number of processing units allocated to the pre-filled instances and the number of processing units allocated to the decoding instances can be equivalent to the total number of processing units included in the processing chip. The specific ratio between the two can be determined by the software / driver based on user settings or based on actual model operation, and this application does not impose any limitations on this. Furthermore, the sum of the bandwidth utilization rate of the processing units allocated to the pre-filled instances across multiple memory units and the bandwidth utilization rate of the processing units allocated to the decoding instances across multiple memory units can be equivalent to the upper limit of the bandwidth of the multiple memory units. The specific ratio between the two can be determined by the software / driver based on user settings or based on actual model operation, and this application does not impose any limitations on this.

[0031] Figure 2 This is a schematic diagram illustrating the instance division of an embodiment of the present invention. For example... Figure 2 As shown, on one hand, embodiments of the present invention can split the processing unit within the processing chip into two parts, and assign the two parts of the processing unit to a pre-filling instance and a decoding instance respectively, so as to realize the configuration of computing power resources 21 (the area covered by gray shadow in computing power resources 21 represents the computing power resource portion configured for the pre-filling instance 231, and the blank area in computing power resources 21 represents the computing power resource portion configured for the decoding instance 232). On the other hand, embodiments of the present invention can use a bandwidth allocation circuit unit set in the processing chip to control the memory access of each processing unit according to the bandwidth weight, so that the bandwidth occupation of each processing unit in the memory unit matches the bandwidth weight of each processing unit, so as to realize the configuration of bandwidth resources 22 (the area covered by gray shadow in bandwidth resources 22 represents the bandwidth resource portion configured for the pre-filling instance 231, and the blank area in bandwidth resources 22 represents the bandwidth resource portion configured for the decoding instance 232). Thus, embodiments of the present invention can obtain pre-filling instance 231 and decoding instance 232 with different computing power resources and different bandwidth resources. The computing resources occupied by the pre-filling instance 231 can be greater than those occupied by the decoding instance 232, and the bandwidth resources occupied by the pre-filling instance 231 can be less than those occupied by the decoding instance 232. Therefore, in this embodiment of the invention, the pre-filling task 241 and the decoding task 242 can be deployed to the pre-filling instance 231 and the decoding instance 232 respectively for execution.

[0032] Figure 3This is a flowchart of a memory access control method according to an embodiment of the present invention. It is intended to be noted that... Figure 3 The execution entity of the memory access control method shown can be the bandwidth allocation circuit unit in the above embodiments. By executing... Figure 3 The memory access control method shown allows the bandwidth allocation circuit unit to control memory access for each processing unit based on bandwidth weights, ensuring that the bandwidth usage of each processing unit in the memory unit matches its bandwidth weight. For example... Figure 3 As shown, the memory access control method may specifically include the following steps: Step S110: Initialize the token bucket of each processing unit.

[0033] Specifically, the bandwidth allocation circuit unit can initialize the token buckets of each processing unit. It should be noted that, for a bandwidth allocation circuit unit implemented in hardware, the token bucket can be a logic circuit capable of recording corresponding counter values ​​and updating the recorded counter values ​​according to instructions. The counter value recorded in each token bucket can be considered as the number of tokens stored in the token bucket.

[0034] Optionally, in step S110, initializing the token bucket of each processing unit may specifically refer to setting an initial number of tokens for the token bucket of each processing unit. The initial number of tokens set for the token bucket of each processing unit can be set by relevant personnel according to actual needs, and this application does not impose any restrictions on this.

[0035] Step S120: Receive memory access requests sent by each of the processing units.

[0036] Specifically, after initializing the token bucket of each processing unit, the bandwidth allocation circuit unit can uniformly receive memory access requests sent by each processing unit. These memory access requests can be triggered by each processing unit and can be used to characterize each processing unit's read / write requests for memory units.

[0037] Step S130: Send each memory access request to the corresponding storage unit according to the token bucket and the bandwidth weight, so as to perform memory access control on each processing unit.

[0038] Specifically, the bandwidth allocation circuit unit can send each memory access request to the corresponding storage unit according to the token bucket and bandwidth weight, so as to perform memory access control for each processing unit.

[0039] Optionally, as an implementation, to control memory access for each processing unit, the bandwidth allocation circuit unit can determine the priority level of each processing unit based on the number of tokens stored in each token bucket on a periodic basis (which can be divided according to a fixed duration or according to event triggering conditions). (Tokens in the token bucket will be consumed during this process). The unit then sends the memory access requests of each processing unit to the corresponding storage unit in a manner that matches the priority level of each processing unit. In this process, the bandwidth weight can be used to indicate the number of tokens that the bandwidth allocation circuit unit replenishes and updates to the token bucket of each processing unit in each period.

[0040] Figure 4 This is a flowchart of a memory access control method according to an embodiment of the present invention. It is intended to be noted that... Figure 4 The execution entity of the memory access control method shown can be the bandwidth allocation circuit unit in the above embodiments. Within each cycle, the bandwidth allocation circuit unit can execute... Figure 4 The memory access control method shown determines the priority level of each processing unit based on the number of tokens stored in each token bucket, and sends the memory access request of each processing unit to the corresponding storage unit in a manner that matches the priority level of each processing unit. For example... Figure 4 As shown, the memory access control method may specifically include the following steps: Step S131: Replenish tokens to the token bucket of each processing unit according to the bandwidth weight.

[0041] Specifically, at the start of the current cycle, the bandwidth allocation circuit unit can first replenish tokens to the token buckets of each processing unit according to the bandwidth weight. Replenishing tokens to the token bucket can be understood as increasing the counter value recorded in the token bucket.

[0042] Alternatively, as a configuration method, the number of tokens replenished to the token bucket of each processing unit in each cycle can be equivalent to the bandwidth weight value of each processing unit.

[0043] Step S132: Determine the high-priority processing unit and low-priority processing unit group based on the number of tokens in each token bucket.

[0044] Specifically, after replenishing tokens, the bandwidth allocation circuit unit can determine high-priority processing units and low-priority processing unit groups based on the number of tokens in each token bucket. A high-priority processing unit can be the processing unit with the most tokens in its token bucket, and a low-priority processing unit group can include all other processing units besides the high-priority processing unit. It should be noted that in this embodiment of the invention, the high-priority processing unit will preferentially occupy bandwidth resources, while the processing units in the low-priority processing unit group will share the remaining bandwidth resources reserved for the high-priority processing unit.

[0045] Step S133: Send the memory access request of the high-priority processing unit to the corresponding storage unit in a manner that occupies the maximum bandwidth, and send the memory access request of each processing unit in the low-priority processing unit group to the corresponding storage unit in a manner that occupies the remaining bandwidth.

[0046] Specifically, after determining the high-priority processing unit and the low-priority processing unit group, the bandwidth allocation circuit unit can send the memory access request of the high-priority processing unit to the corresponding memory unit in a manner that occupies the maximum bandwidth, and send the memory access request of each processing unit in the low-priority processing unit group to the corresponding memory unit in a manner that occupies the remaining bandwidth.

[0047] To clarify, sending memory access requests from high-priority processing units to their corresponding storage units using maximum bandwidth can specifically mean sending these requests consecutively to the corresponding storage units. Sending memory access requests from each processing unit in the low-priority processing unit group to their respective storage units using remaining bandwidth can specifically mean sending these requests only to their respective storage units when there are no pending memory access requests from high-priority processing units and the current bandwidth is not fully utilized.

[0048] It should be noted that, in each cycle, once the memory access request sending method for each processing unit is determined, the bandwidth allocation circuit unit will send the memory access request for each processing unit using the determined memory access request sending method until the next access control cycle arrives.

[0049] Step S134: Deduct tokens from the token bucket of the high-priority processing unit according to the bandwidth weight.

[0050] Specifically, after determining the high-priority processing units and low-priority processing unit groups, the bandwidth allocation circuit unit can also deduct tokens from the token bucket of the high-priority processing units according to the bandwidth weight. Deducting tokens from the token bucket can be understood as lowering the counter value recorded in the token bucket.

[0051] Alternatively, as a configuration method, the number of tokens deducted from the token bucket of a high-priority processing unit can be equal to the sum of the bandwidth weight values ​​of the processing units.

[0052] To facilitate understanding, the memory access control flow implemented by the bandwidth allocation circuit unit will be explained below with a specific example. Illustratively, in this example, the processing chip includes processing units A, B, C, and D. Processing unit A is assigned to the pre-filled instance, while processing units B, C, and D are assigned to the decoding instance. The number of tokens in the token buckets of processing units A, B, C, and D is set to 0 during the token bucket initialization operation. Furthermore, the bandwidth weights of processing units A, B, C, and D are 3, 1, 1, and 1, respectively.

[0053] Figure 5 This is a schematic diagram of the memory access control flow according to an embodiment of the present invention. It is intended to illustrate that... Figure 5 Table 51 shown illustrates the token bucket updates and priority settings for processing units A, B, C, and D by the bandwidth allocation circuit unit across multiple access control cycles. Figure 5 As shown, in the first access control cycle, the bandwidth allocation circuit unit can replenish the token buckets of processing units A, B, C, and D with tokens of 3, 1, 1, and 1 respectively, so that the number of tokens in the token buckets of processing units A, B, C, and D are 3, 1, 1, and 1 respectively. Furthermore, the bandwidth allocation circuit unit can designate processing unit A as a high-priority processing unit and processing units B, C, and D as a low-priority processing unit group, and send the memory access request of processing unit A to the corresponding storage unit in a manner that utilizes the maximum bandwidth, and send the memory access requests of processing units B, C, and D to the corresponding storage units in a manner that utilizes the remaining bandwidth. Simultaneously, the bandwidth allocation circuit unit can replenish the token bucket of processing unit A with 6 tokens, so that the number of tokens in the token buckets of processing units A, B, C, and D are -3, 1, 1, and 1 respectively.

[0054] In the second access control cycle, the bandwidth allocation circuit unit can replenish the token buckets of processing units A, B, C, and D with 3, 1, 1, and 1 tokens respectively, so that the number of tokens in the token buckets of processing units A, B, C, and D are 0, 2, 2, and 2 respectively. Furthermore, the bandwidth allocation circuit unit can designate processing unit B as a high-priority processing unit and processing units A, C, and D as a low-priority processing unit group. It then sends the memory access request of processing unit B to the corresponding storage unit in a manner that utilizes the maximum bandwidth, and sends the memory access requests of processing units A, C, and D to the corresponding storage units in a manner that utilizes the remaining bandwidth. Simultaneously, the bandwidth allocation circuit unit can replenish the token bucket of processing unit B with 6 tokens, so that the number of tokens in the token buckets of processing units A, B, C, and D are 0, -4, 2, and 2 respectively. It should be noted that, as one implementation, when the number of tokens in the token buckets of multiple processing units is the same, the bandwidth allocation circuit unit can randomly select one processing unit from among the multiple processing units as the high-priority processing unit.

[0055] In the third access control cycle, the bandwidth allocation circuit unit can replenish the token buckets of processing units A, B, C, and D with tokens of 3, 1, 1, and 1 respectively, so that the token buckets of processing units A, B, C, and D contain 3, -3, 3, and 3 tokens respectively. Furthermore, the bandwidth allocation circuit unit can designate processing unit C as a high-priority processing unit and processing units A, B, and D as a low-priority processing unit group. It then sends the memory access request of processing unit C to the corresponding storage unit in a manner that utilizes the maximum bandwidth, and sends the memory access requests of processing units A, B, and D to the corresponding storage units in a manner that utilizes the remaining bandwidth. Simultaneously, the bandwidth allocation circuit unit can replenish the token bucket of processing unit C with 6 tokens, so that the token buckets of processing units A, B, C, and D contain 3, -3, -3, and 3 tokens respectively.

[0056] In the fourth access control cycle, the bandwidth allocation circuit unit can replenish the token buckets of processing units A, B, C, and D with 3, 1, 1, and 1 tokens respectively, so that the token buckets of processing units A, B, C, and D contain 6, -2, -2, and 4 tokens respectively. Furthermore, the bandwidth allocation circuit unit can designate processing unit A as a high-priority processing unit and processing units B, C, and D as a low-priority processing unit group. It then sends memory access requests from processing unit A to the corresponding storage unit using maximum bandwidth, and sends memory access requests from processing units B, C, and D to the corresponding storage units using the remaining bandwidth. Simultaneously, the bandwidth allocation circuit unit can replenish the token bucket of processing unit A with 6 tokens, so that the token buckets of processing units A, B, C, and D contain 0, -2, -2, and 4 tokens respectively.

[0057] It should be noted that in each subsequent access control cycle, the bandwidth allocation circuit unit executes the above process to determine the priority level of each processing unit based on the number of tokens stored in each token bucket (tokens in the token bucket are consumed in this process), and sends the memory access requests of each processing unit to the corresponding storage unit in a manner that matches the priority level of each processing unit. Therefore, further examples will not be provided here. Thus, the embodiments of the present invention can ensure that the bandwidth usage of each processing unit on the storage unit is matched with the bandwidth weight of each processing unit.

[0058] It should be noted that the memory access control method given above is only for illustration. In actual application, the bandwidth allocation circuit unit can also use other methods to control the memory access of each processing unit based on the bandwidth weight of each processing unit and the token bucket, so that the bandwidth occupation of each processing unit in the memory unit matches the bandwidth weight of each processing unit. This application does not impose any restrictions on this.

[0059] Optionally, in addition to multiple processing units, multiple storage units, bandwidth allocation circuit units, and packet on-chip networks, the processing chip may also include a first logic module and a second logic module. The first logic module can be used to configure computing resources for pre-filled instances and decoding instances. The second logic module can configure bandwidth resources for pre-filled instances and decoding instances.

[0060] Figure 6 This is a flowchart illustrating a computing resource allocation method according to an embodiment of the present invention. It is intended to be noted that... Figure 6 The execution entity of the computing resource allocation method shown can be the first logical module. Through execution... Figure 6 The computing resource configuration method shown herein allows the first logic module to configure computing resources for pre-filled instances and decoding instances. For example... Figure 6 As shown, the computing power resource allocation method may specifically include the following steps: Step S211: Receive computing resource configuration information.

[0061] Specifically, the first logic module can receive computing resource configuration information. This computing resource configuration information can be used to characterize the computing resource configuration requirements for pre-filled instances and decoding instances.

[0062] Optionally, in step S211, the computing resource configuration information can be triggered by software / driver, and it can be determined by software / driver based on user settings or based on the actual model operation. This application does not impose any restrictions on this.

[0063] Step S212: Determine the computing resource allocation ratio based on the computing resource configuration information.

[0064] Specifically, after receiving the computing power resource allocation information, the first logic module can determine the computing power resource allocation ratio based on the computing power resource allocation information.

[0065] Optionally, as one implementation, the computing resource allocation information may include a computing resource allocation ratio. In step S212, the first logic module can parse and determine the computing resource allocation ratio from the computing resource allocation information. It should be noted that the computing resource allocation information can also represent the computing resource allocation requirements for pre-filled instances and decoding instances in other ways. In this case, the first logic module can determine the computing resource allocation ratio using a matching method, and this application does not specifically limit this.

[0066] Step S213: According to the computing power resource allocation ratio, each processing unit is allocated to the pre-filled instance or the decoding instance respectively.

[0067] Specifically, after determining the computing power resource allocation ratio, the first logic module can allocate each processing unit to a pre-filled instance or a decoding instance according to the computing power resource allocation ratio, so that the ratio of the number of processing units allocated to the pre-filled instance to the number of processing units allocated to the decoding instance matches the computing power resource allocation ratio.

[0068] Figure 7 This is a flowchart illustrating a bandwidth resource configuration method according to an embodiment of the present invention. It is intended to be noted that... Figure 7 The execution entity of the bandwidth resource configuration method shown can be a second logical module. Through execution... Figure 7 The bandwidth resource configuration method shown herein allows the second logic module to configure bandwidth resources for pre-filled instances and decoding instances. For example... Figure 7 As shown, the bandwidth resource configuration method may specifically include the following steps: Step S221: Receive bandwidth resource configuration information.

[0069] Specifically, the second logic module can receive bandwidth resource configuration information. This bandwidth resource configuration information can be used to characterize the bandwidth resource configuration requirements for pre-filled instances and decoding instances. The bandwidth resource configuration information can be triggered by software / drivers, and can be determined by the software / drivers based on user settings or actual model operation; this application does not impose any restrictions on this.

[0070] Step S222: Determine the bandwidth weight of each processing unit based on the bandwidth resource configuration information.

[0071] Specifically, upon receiving bandwidth resource configuration information, the second logic module can determine the bandwidth weight of each processing unit based on the bandwidth resource configuration information.

[0072] Figure 8 This is a flowchart of a bandwidth weight determination method according to an embodiment of the present invention. By executing... Figure 8 The bandwidth weight determination method shown allows the second logic module to determine the bandwidth weight of each processing unit based on bandwidth resource configuration information, thus implementing step S222 above. Figure 8 As shown, the bandwidth weight determination method may specifically include the following steps: Step S2221: Determine the pre-fill bandwidth weight of the pre-filled instance and the decoding bandwidth weight of the decoding instance based on the bandwidth resource configuration information.

[0073] Specifically, the second logic module can determine the pre-fill bandwidth weight of the pre-filled instance and the decoding bandwidth weight of the decoding instance based on the bandwidth resource configuration information.

[0074] Optionally, as one implementation, the bandwidth resource configuration information may include pre-filled bandwidth weights and decoding bandwidth weights. In step S2221, the second logic module can parse and determine the pre-filled bandwidth weights and decoding bandwidth from the bandwidth resource configuration information. It should be noted that the bandwidth resource configuration information can also represent the bandwidth resource configuration requirements for the pre-filled instance and the decoding instance in other ways. In this case, the second logic module can determine the pre-filled bandwidth weights and decoding bandwidth using a matching method, and this application does not specifically limit this.

[0075] Step S2222: For the pre-filled instance, the pre-filled bandwidth weight is evenly distributed according to the number of processing units assigned to the pre-filled instance to determine the bandwidth weight of each processing unit in the pre-filled instance.

[0076] Specifically, after determining the pre-filled bandwidth weight and the decoding bandwidth weight, for a pre-filled instance, the second logic module can equally distribute the pre-filled bandwidth weight according to the number of processing units assigned to the pre-filled instance, so as to determine the bandwidth weight of each processing unit in the pre-filled instance.

[0077] To illustrate, assume that the number of processing units assigned to a pre-filled instance is 1, and the pre-filled bandwidth weight is 3. Then, the second logic module can evenly distribute the pre-filled bandwidth weight according to the number of processing units assigned to the pre-filled instance to determine that the bandwidth weight of the processing unit in the pre-filled instance is 3.

[0078] Step S2223: For the decoding instance, the decoding bandwidth weight is evenly distributed according to the number of processing units assigned to the decoding instance to determine the bandwidth weight of each processing unit in the decoding instance.

[0079] Specifically, after determining the pre-filled bandwidth weight and the decoding bandwidth weight, for a decoding instance, the second logic module can equally distribute the decoding bandwidth weight according to the number of processing units assigned to the decoding instance, so as to determine the bandwidth weight of each processing unit in the decoding instance.

[0080] To illustrate, assume that the number of processing units assigned to the decoding instance is 3, and the pre-filled bandwidth weight is 3. Then, the second logic module can evenly distribute the pre-filled bandwidth weight according to the number of processing units assigned to the decoding instance, so as to determine that the bandwidth weight of each processing unit in the decoding instance is 1.

[0081] It should be noted that when setting bandwidth weights, the bandwidth weight of each instance can be set to an integer multiple of the number of processing units within each instance, so as to ensure that the computing resources occupied by the processing units within each instance can be evenly distributed.

[0082] Step S223: Store each bandwidth weight into its corresponding bandwidth register.

[0083] Specifically, after determining the bandwidth weight of each processing unit, the first logic module can store each bandwidth weight into the corresponding bandwidth register.

[0084] This invention assigns each processing unit in a processing chip to either a pre-filling instance or a decoding instance. Processing units assigned to pre-filling instances handle pre-filling tasks, while those assigned to decoding instances handle decoding tasks. A weight register is set in each processing unit, and a bandwidth allocation circuit unit is included in the processing chip. This bandwidth allocation circuit unit controls memory access for each processing unit based on its bandwidth weight, ensuring that the bandwidth usage of each processing unit matches its bandwidth weight. Therefore, this invention can separately configure computing and bandwidth resources for pre-filling and decoding instances, thus supporting the separation of pre-filling and decoding instances within a single chip.

[0085] Figure 9 This is a schematic diagram of the processing system according to an embodiment of the present invention. Figure 9As shown, the processing system may include a processing chip 91 as described in the above embodiments. The processing chip 91 can be understood as an integrated circuit specifically designed for performing computational, logic control, and data processing tasks. Optionally, the number of processing chips included in the processing system may be single or multiple, and this application does not impose any limitation on this. Furthermore, in addition to the processing chip, the electronic device may also include a memory 92. The memory 92 differs from the storage units within the processing chip 91. The memory 92 has a larger storage capacity and is used to store the complete program required by the processing chip or all complete parameters of a deployed large language model, etc. During actual operation, the processing chip 91 can load the required data from the memory 92 into its own internal storage units as needed. Optionally, the processing system may be a graphics processor, a central processing unit, a tensor processor, an embedded neural network processor, a digital signal processor, a language processor, etc., and this application does not impose any limitation on this.

[0086] Figure 10 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Figure 10 As shown, the electronic device may include the processing system described in the above embodiments. Optionally, the number of processing systems included in the electronic device may be single or multiple, and this application does not limit this. In addition to the processing system, the electronic device may also include a display device 102, a communication component 103, and an input / output (I / O) device 104, etc. The display device 102 can be used to implement external output of the electronic device. The communication component 103 can be used to implement external communication of the electronic device. The input / output (I / O) device 104 can be used to implement control or command input of the electronic device. Optionally, the input / output (I / O) device 104 may be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Furthermore, as a connection method, the above-mentioned processing system and functional components can be interconnected via a bus to realize instruction interaction and data transmission between them.

[0087] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A processing chip, characterized in that, The processing chip includes: Multiple processing units are provided, wherein each processing unit is assigned to a pre-fill instance or a decoding instance. The processing unit assigned to the pre-fill instance is used to process the pre-fill task, and the processing unit assigned to the decoding instance is used to process the decoding task. Each processing unit includes a weight register for storing the bandwidth weight of the processing unit. Multiple storage units are provided for access by the multiple processing units; A bandwidth allocation circuit unit is disposed between each of the processing units and each of the storage units. The bandwidth allocation circuit unit is configured to perform memory access control on each of the processing units according to the bandwidth weights, so that the bandwidth usage of each processing unit on the storage units matches the bandwidth weight of each processing unit; and... A packet-on-chip network is used to realize communication connections between each of the processing units and the bandwidth allocation circuit unit, and to realize communication connections between the bandwidth allocation circuit unit and each of the storage units.

2. The processing chip according to claim 1, characterized in that, The number of processing units assigned to the pre-filled instance is greater than the number of processing units assigned to the decoding instance, and the bandwidth utilization rate of the processing units assigned to the pre-filled instance to the plurality of storage units is less than the bandwidth utilization rate of the processing units assigned to the decoding instance to the plurality of storage units.

3. The processing chip according to claim 1, characterized in that, The bandwidth allocation circuit unit is specifically configured as follows: Initialize the token bucket of each of the processing units; Receive memory access requests sent by each of the processing units; Each memory access request is sent to the corresponding storage unit according to the token bucket and the bandwidth weight, so as to perform memory access control on each processing unit.

4. The processing chip according to claim 3, characterized in that, The bandwidth allocation circuit unit is specifically configured as follows: The token bucket of each processing unit is replenished with tokens according to the bandwidth weight; High-priority processing units and low-priority processing unit groups are determined based on the number of tokens in each token bucket. The high-priority processing unit is the processing unit that contains the most tokens in the token bucket, and the low-priority processing unit group includes other processing units besides the high-priority processing unit. The memory access requests of the high-priority processing unit are sent to the corresponding storage unit in a manner that utilizes the maximum bandwidth, and the memory access requests of each processing unit in the low-priority processing unit group are sent to the corresponding storage unit in a manner that utilizes the remaining bandwidth. Tokens are deducted from the token bucket of the high-priority processing unit according to the bandwidth weight.

5. The processing chip according to claim 4, characterized in that, The number of tokens replenished to the token bucket of each processing unit is equal to the bandwidth weight value of each processing unit, and the number of tokens deducted from the token bucket of the high-priority processing unit is equal to the sum of the bandwidth weight values ​​of each processing unit.

6. The processing chip according to claim 1, characterized in that, The processing chip also includes: The first logic module is configured to receive bandwidth resource configuration information, determine the bandwidth weight of each processing unit according to the bandwidth resource configuration information, and store each bandwidth weight into the corresponding bandwidth register.

7. The processing chip according to claim 6, characterized in that, The first logic module is specifically configured as follows: The pre-fill bandwidth weight of the pre-filled instance and the decoding bandwidth weight of the decoding instance are determined based on the bandwidth resource configuration information. For the pre-filled instance, the pre-filled bandwidth weight is evenly distributed according to the number of processing units assigned to the pre-filled instance to determine the bandwidth weight of each processing unit in the pre-filled instance. For the decoding instance, the decoding bandwidth weight is evenly distributed according to the number of processing units assigned to the decoding instance to determine the bandwidth weight of each processing unit in the decoding instance.

8. The processing chip according to claim 1, characterized in that, The processing chip also includes: The second logic module is configured to receive computing power resource configuration information, determine the computing power resource allocation ratio according to the computing power resource configuration information, and allocate each of the processing units to the pre-filled instance or the decoding instance according to the computing power resource allocation ratio.

9. The processing chip according to claim 1, characterized in that, The packet-on-chip network includes: A first on-chip network is disposed between the plurality of processing units and the bandwidth allocation circuit unit, and the first on-chip network is used to realize the communication connection between each of the processing units and the bandwidth allocation circuit unit; The second on-chip network is disposed between the bandwidth allocation circuit unit and the plurality of storage units, and the first on-chip network is used to realize the communication connection between the bandwidth allocation circuit unit and each of the storage units.

10. The processing chip according to claim 1, characterized in that, The processing chip is a graphics processor, the processing unit is a streaming multiprocessor, and the storage unit is a last-level cache.

11. A processing system, characterized in that, The processing system includes the processing chip as described in any one of claims 1-10.

12. An electronic device, characterized in that, The electronic device includes the processing system as described in claim 11.

Images