Artificial intelligence chip and operating method thereof

Abstract

The application provides an artificial intelligence chip and an operation method thereof. The AI chip includes an AI core, a flow control circuit, an arbitrator, and a cache. The arbitrator is coupled between the AI core and the cache. The flow control circuit is coupled between the AI core and the arbitrator. The flow control circuit counts back pressure of feedback data in a first feedback data path from the cache to the AI core through the arbitrator. The flow control circuit dynamically decides whether to block an access request handshake between the AI core and the cache according to the back pressure of the feedback data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of integrated circuits, and in particular, to an artificial intelligence (AI) chip and an operation method thereof. BACKGROUND

[0002] In a graphics processing unit (GPU) chip, a General-purpose GPU (GPGPU) chip, or an artificial intelligence (AI) chip, a uniform memory access (UMA) access mode is usually adopted. Multiple AI cores can access a cache. In actual use, the workloads of different AI cores are not necessarily consistent, which may cause inconsistent bandwidths when multiple AI cores access the cache. AI cores running low-bandwidth programs will back up the entire system, causing the bandwidth of the entire system to decrease. How to more efficiently use multiple AI cores is one of the many technical issues in the field. SUMMARY

[0003] The present application is directed to an artificial intelligence (AI) chip and an operation method thereof, to balance the access bandwidth of different AI cores to a cache.

[0004] In an embodiment according to the present application, the artificial intelligence chip includes a first artificial intelligence core, a flow control circuit, an arbiter, and a cache. The arbiter is coupled to the cache. The flow control circuit is coupled between the first artificial intelligence core and the arbiter. The flow control circuit counts a back pressure condition of feedback data in a first feedback data path from the cache to the first artificial intelligence core through the arbiter. The flow control circuit dynamically decides whether to block a handshake of an access request between the first artificial intelligence core and the cache according to the back pressure condition of the feedback data.

[0005] In an embodiment according to the present application, the operation method includes: counting, by a flow control circuit of an artificial intelligence chip, a back pressure condition of feedback data in a first feedback data path from a cache of the artificial intelligence chip to a first artificial intelligence core of the artificial intelligence chip through an arbiter of the artificial intelligence chip; and dynamically deciding, by the flow control circuit, whether to block a handshake of an access request between the first artificial intelligence core and the cache according to the back pressure condition of the feedback data.

[0006] Based on the above, the flow control circuit can detect the backpressure condition of the feedback data in the first feedback data path between the cache and the first artificial intelligence core. For example, from the time when the cache sends a valid signal for the feedback data to the time when the first artificial intelligence core returns a ready signal, the long delay time can reflect the backpressure condition of the first feedback data path between the cache and the first artificial intelligence core. The flow control circuit can count one or more long delay times of the first feedback data path between the cache and the first artificial intelligence core to determine the backpressure condition of the feedback data. In order to avoid the serious backpressure condition of the first artificial intelligence core to lower the efficiency of the entire system, the flow control circuit can temporarily block the access request handshake between the first artificial intelligence core and the cache when the backpressure condition of the feedback data of the first artificial intelligence core is serious. When the backpressure condition of the feedback data of the first artificial intelligence core is improved, the flow control circuit can restore the access request handshake between the first artificial intelligence core and the cache. Therefore, the flow control circuit can balance the access bandwidth of different artificial intelligence cores to the cache.

[0007] In order to make the above features and advantages of the present application more apparent, specific embodiments are described below in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a circuit block diagram of an artificial intelligence (AI) chip according to an embodiment of the present application.

[0009] Figure 2 is a flowchart of an operation method of an artificial intelligence chip according to an embodiment of the present application.

[0010] Figure 3 is a circuit block diagram of a flow control circuit according to an embodiment of the present application.

[0011] Figure 4 is a circuit block diagram of a flow control circuit according to an embodiment of the present application.

[0012] Figure 5 is a circuit block diagram of a data path backpressure counting circuit according to an embodiment of the present application.

[0013] Figure 6 is a circuit block diagram of an AI chip according to another embodiment of the present application.

[0014] Figure 7 is a circuit block diagram of a flow control circuit according to an embodiment of the present application.

[0015] Figure 8 is shown in accordance with an embodiment of the present application, a circuit block diagram of a flow controller.

[0016] Figure 9 is shown in accordance with an embodiment of the present application, a circuit block diagram of a data path backpressure counting circuit.

[0017] BRIEF DESCRIPTION OF DRAWINGS

[0018] 100: a first artificial intelligence chip,

[0019] 600: a second artificial intelligence chip,

[0020] 110_1: a first_1 artificial intelligence core,

[0021] 110_N: a first_N artificial intelligence core,

[0022] 610_1: a second_1 artificial intelligence core,

[0023] 610_N: a second_N artificial intelligence core,

[0024] 120: a first flow control circuit,

[0025] 620: a second flow control circuit,

[0026] 121_1: a first_1 flow controller,

[0027] 121_N: a first_N flow controller,

[0028] 621_1: a second_1 flow controller,

[0029] 621_N: a second_N flow controller,

[0030] 130: a first arbiter,

[0031] 630: a second arbiter,

[0032] 140: a first cache,

[0033] 640: a second cache,

[0034] 150: a first main memory,

[0035] 650: a second main memory,

[0036] 410: a first data path backpressure counting circuit,

[0037] 810: a second data path backpressure counting circuit,

[0038] 411: a first data path backpressure counter,

[0039] 811: 2nd data path back pressure counter,

[0040] 412: 1st_1 register,

[0041] 413: 1st_2 register,

[0042] 813: 2nd_2 register,

[0043] 812: 2nd_1 register,

[0044] 414: 1st calculator,

[0045] 814: 2nd calculator,

[0046] 420: 1st access request path controller,

[0047] 820: 2nd access request path controller,

[0048] 501: 1st current back pressure coefficient,

[0049] 901: 2nd current back pressure coefficient,

[0050] 502: 1st_1 previous back pressure coefficient,

[0051] 503: 1st_2 previous back pressure coefficient,

[0052] 902: 2nd_1 previous back pressure coefficient,

[0053] 903: 2nd_2 previous back pressure coefficient,

[0054] 601: task allocation engine,

[0055] 622: central flow controller,

[0056] BP4: back pressure coefficient,

[0057] BP71: 1st back pressure coefficient,

[0058] BP72: 2nd back pressure coefficient,

[0059] BP73: 3rd back pressure coefficient,

[0060] BP74: 4th back pressure coefficient,

[0061] TD6: task allocation information. DETAILED DESCRIPTION

[0062] Reference will now be made in detail to exemplary embodiments of the application, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used in the different drawings and the description to refer to the same or like parts.

[0063] The term "coupled" or "connected" used in the entire specification (including claims) can refer to any direct or indirect connection. For example, if it is described in the text that a first device is coupled (or connected) to a second device, it should be interpreted that the first device can be directly connected to the second device, or the first device can be indirectly connected to the second device through other devices or some connection means. The terms "first", "second", and the like used in the entire specification (including claims) are used to name components or to distinguish different embodiments or ranges, and are not used to limit the upper or lower limit of the number of components, nor to limit the order of components. In addition, wherever possible, the same reference numbers are used for components / means / steps in the drawings and embodiments to represent the same or similar parts. Components / means / steps with the same reference numbers or using the same terms in different embodiments can be referred to each other in the relevant description. It should be understood that the features of the following embodiments can be combined with each other. For example, the features of the second embodiment can be combined with the features of the first embodiment. Those skilled in the art can select appropriate combinations of features according to actual design needs.

[0064] An operation device such as an artificial intelligence (AI) chip can provide tremendous computing power. The tremendous computing power of an AI chip is derived from a large number of hardware cores inside. An AI chip usually contains multiple programmable processors, such as a Stream Processor Cluster (SPC). Each programmable processor usually contains multiple compute units (CUs, or computing cores). Each computing core usually contains multiple execution units (EUs, or execution cores). The execution cores include at least one of a tensor core (Tcore), an integer (INT) core, a floating point (FP) core, and a vector core (Vcore), for example. By programming these computing cores, an AI chip can support general-purpose computing, scientific computing, neural network computing, or other computing.

[0065] Figure 1 is a circuit block schematic diagram of a first artificial intelligence (AI) chip 100 according to an embodiment of the present application. In Figure 1 In the embodiment shown, the first artificial intelligence chip 100 includes multiple artificial intelligence (AI) cores (such as Figure 1The diagram shows AI cores 1_1 to 1_N (AI cores 110_1, ..., 1_N). The number N of AI cores 1_1 to 1_N can be determined according to the actual design and application. Each AI core 1_1 to 1_N typically contains multiple execution units (EUs, or execution cores, not shown) and shared memory (not shown). Based on the actual design and application, the execution cores include at least one of integer cores, floating-point cores, and vector cores. Different execution cores within the same AI core can exchange data with each other through shared memory.

[0066] Figure 1 The first artificial intelligence chip 100 shown also includes a first flow control circuit 120, a first arbitrator 130, a first cache 140, and a first main memory 150. The first cache 140 is coupled between the first AI cores 110_1 to 1_N AI cores 110_N and the first main memory 150. The first arbitrator 130 is coupled between the first AI cores 110_1 to 1_N AI cores 110_N and the first cache 140. The first flow control circuit 120 is coupled between the first AI cores 110_1 to 1_N AI cores 110_N and the first arbitrator 130. Depending on the design, in some embodiments, at least one of the first flow control circuit 120 and the first arbitrator 130 may be implemented as hardware circuitry. In other embodiments, at least one of the first flow control circuit 120 and the first arbitrator 130 may be implemented as a combination of hardware, firmware, and software (i.e., programs).

[0067] In hardware form, at least one of the first flow control circuit 120 and the first arbiter 130 can be implemented as logic circuitry on an integrated circuit. For example, the functions of at least one of the first flow control circuit 120 and the first arbiter 130 can be implemented in various logic blocks, modules, and circuits of one or more hardware controllers, microcontrollers, hardware processors, microprocessors, application-specific integrated circuits (ASICs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), central processing units (CPUs), or other processing units. The functions of at least one of the first flow control circuit 120 and the first arbiter 130 can be implemented in various logic blocks, modules, and circuits of hardware circuitry, such as integrated circuitry, using hardware description languages (e.g., Verilog HDL or VHDL) or other suitable programming languages.

[0068] In software form or firmware form, the functions of at least one of the first flow control circuit 120 and the first arbiter 130 can be implemented as programming codes. For example, at least one of the first flow control circuit 120 and the first arbiter 130 can be implemented using general programming languages (e.g., C, C++, or assembly language) or other suitable programming languages. The programming codes can be recorded / stored in a "non-transitory machine-readable storage medium." In some embodiments, the non-transitory machine-readable storage medium includes, for example, semiconductor memories and / or storage devices. An electronic device (e.g., a CPU, a hardware controller, a microcontroller, a hardware processor, or a microprocessor) can read and execute the programming codes from the non-transitory machine-readable storage medium to implement the functions of at least one of the first flow control circuit 120 and the first arbiter 130.

[0069] Figure 2is a flowchart of an operating method of an artificial intelligence chip according to an embodiment of the present application. Please refer to Figure 1 With Figure 2 In step S210, the first flow control circuit 120 counts the back pressure condition of the feedback data in the feedback data path from the first cache 140 to any one of the first_1 artificial intelligence core 110_1 to the first_N artificial intelligence core 110_N through the first arbiter 130. For the convenience of explanation, the first_1 artificial intelligence core 110_1 is taken as an example of illustration. The rest of the AI cores (for example, the first_N artificial intelligence core 110_N) can refer to the relevant description of the first_1 artificial intelligence core 110_1 and be analogized, so it is not described in detail. From the first cache 140 sending a valid signal (valid signal, indicating that the first cache 140 is ready for data) to the first_1 artificial intelligence core 110_1 returning a ready signal (ready signal, indicating that the first_1 artificial intelligence core 110_1 is ready to receive), the delay time can reflect the back pressure condition of the feedback data path between the first cache 140 and the first_1 artificial intelligence core 110_1. The first flow control circuit 120 can count one or more delay times of the feedback data path between the first cache 140 and the first_1 artificial intelligence core 110_1 to determine the back pressure condition of the feedback data.

[0070] In step S220, the first flow control circuit 120 dynamically decides whether to block the access request handshake between any one of the first_1 artificial intelligence core 110_1 to the first_N artificial intelligence core 110_N and the first cache 140 according to the back pressure condition of the feedback data. For the convenience of explanation, the first_1 artificial intelligence core 110_1 is taken as an example of illustration. The rest of the AI cores (for example, the first_N artificial intelligence core 110_N) can refer to the relevant description of the first_1 artificial intelligence core 110_1 and be analogized, so it is not described in detail. The blocking of the access request handshake between the first_1 artificial intelligence core 110_1 and the first cache 140 means that the density of the access request sent by the first_1 artificial intelligence core 110_1 to the first cache 140 is dynamically reduced.

[0071] In summary, the first flow control circuit 120 can detect the backpressure of feedback data in the feedback data path between the first cache 140 and any of the first AI cores 1_1 to 1_N AI cores 110_N. Taking the first AI core 110_1 as an example, to avoid the severe backpressure of the first AI core 110_1 reducing the efficiency of the entire system, when the backpressure of the feedback data of the first AI core 110_1 is severe, the first flow control circuit 120 can temporarily block the access request handshake between the first AI core 110_1 and the first cache 140 (that is, temporarily stop the access requests issued by the first AI core 110_1 to the first cache 140). When the backpressure situation of the feedback data of the first AI core 110_1 improves, the first flow control circuit 120 can dynamically restore the access request handshake between the first AI core 110_1 and the first cache 140 (that is, restore the access request issued by the first AI core 110_1 to the first cache 140). Therefore, the first flow control circuit 120 can dynamically balance the access bandwidth of different AI cores 110_1 to 1N AI cores 110_N to the first cache 140.

[0072] Figure 3 This is a circuit block diagram of the first flow control circuit 120, drawn according to an embodiment of the present invention. Figure 3 The first flow control circuit 120 shown can be used as Figure 1 This is one of many implementation examples of the first flow control circuit 120 shown. Figure 3 The first AI cores 110_1 to 110_N, the first flow control circuit 120, and the first arbitrator 130 shown can be referenced. Figure 1 The relevant explanations are omitted here. Figure 3 In the illustrated embodiment, the first flow control circuit 120 includes multiple flow controllers (e.g., Figure 3 The diagram shows the first flow controller 121_1, ..., the first flow controller 121_N. For ease of explanation, the first AI core 110_1 and the first flow controller 121_1 are used as examples below. The other AI cores (e.g., the first AI core 110_N) and the other flow controllers (e.g., the first flow controller 121_N) can be deduced by referring to the relevant descriptions of the first AI core 110_1 and the first flow controller 121_1, and therefore will not be elaborated upon.

[0073] The 1st_1 flow controller 121_1 is coupled between the 1st_1 artificial intelligence core 110_1 and the 1st arbiter 130. The 1st_1 flow controller 121_1 counts backpressure conditions of feedback data in a feedback data path from the 1st cache 140 through the 1st arbiter 130 to the 1st_1 artificial intelligence core 110_1. According to the backpressure conditions of feedback data in the feedback data path from the 1st cache 140 to the 1st_1 artificial intelligence core 110_1, the 1st_1 flow controller 121_1 dynamically decides whether to block access request handshakes between the 1st_1 artificial intelligence core 110_1 and the 1st cache 140.

[0074] Figure 4 Fig. 1 1 shows a circuit block diagram of the 1st_1 flow controller 121_1 according to an embodiment of the present application. Figure 4 The 1st_1 flow controller 121_1 shown in Fig. 1 1 can be implemented as Figure 3 The 1st_1 flow controller 121_1 shown in Fig. 1 1 can be implemented as one of the embodiments. Figure 4 The 1st_1 artificial intelligence core 110_1, the 1st_1 flow controller 121_1 and the 1st arbiter 130 shown in Fig. 1 can refer to the related descriptions of Figure 3 The 1st_1 artificial intelligence core 110_1, the 1st_1 flow controller 121_1 and the 1st arbiter 130 shown in Fig. 1 can refer to the related descriptions of Figure 4 In the embodiment shown in Fig. 1 1, the 1st_1 flow controller 121_1 includes a 1st data path backpressure counting circuit 410 and a 1st access request path controller 420.

[0075] The 1st data path backpressure counting circuit 410 is coupled between the 1st_1 artificial intelligence core 110_1 and the 1st arbiter 130. The 1st data path backpressure counting circuit 410 counts backpressure conditions of feedback data in a feedback data path from the 1st cache 140 through the 1st arbiter 130 to the 1st_1 artificial intelligence core 110_1 to generate a backpressure coefficient BP4. The 1st access request path controller 420 is coupled between the 1st_1 artificial intelligence core 110_1 and the 1st arbiter 130. The 1st access request path controller 420 is coupled to the 1st data path backpressure counting circuit 410 to receive the backpressure coefficient BP4. The 1st access request path controller 420 dynamically decides whether to block access request handshakes between the 1st_1 artificial intelligence core 110_1 and the 1st cache 140 according to the backpressure coefficient BP4.

[0076] Figure 5 Fig. 12 shows a circuit block diagram of the 1st data path backpressure counting circuit 410 according to an embodiment of the present application. Figure 5The first data path backpressure counting circuit 410 can be implemented as Figure 4 The first data path backpressure counting circuit 410 can be implemented as Figure 5 The first_1 artificial intelligence core 110_1, the first data path backpressure counting circuit 410, the first access request path controller 420, and the first arbiter 130 can refer to the related descriptions of Figure 4 The first data path backpressure counting circuit 410 can be implemented as Figure 5 In the embodiment shown, the first data path backpressure counting circuit 410 includes a first data path backpressure counter 411, a first_1 register 412, a first_2 register 413, and a first calculator 414.

[0077] The first data path backpressure counter 411 is coupled between the first_1 artificial intelligence core 110_1 and the first arbiter 130. The first data path backpressure counter 411 counts backpressure conditions of feedback data in a feedback data path from the first cache 140 to the first_1 artificial intelligence core 110_1 through the first arbiter 130 to generate a first current backpressure coefficient 501. For example, but not limited to, the first data path backpressure counter 411 accumulates delay time long of different feedback data in a preset time window. The delay time long of feedback data is from the first cache 140 sending a valid signal (valid signal, indicating that the first cache 140 is ready for data) to the first_1 artificial intelligence core 110_1 returning a ready signal (ready signal, indicating that the first_1 artificial intelligence core 110_1 is ready to receive) until the end of this time long. The delay time long can reflect the backpressure condition of the feedback data path between the first cache 140 and the first_1 artificial intelligence core 110_1. The first data path backpressure counter 411 accumulates the delay time long of different feedback data in a preset time window to obtain a total delay time long, and uses the total delay time long as the first current backpressure coefficient 501. The preset time window can be determined according to actual design. For example, the preset time window can be configured by a register to be 100 cycles or other time long.

[0078] Register 412 (1_1) and Calculator 414 (1_2) are coupled to the first data path backpressure counter 411 to receive the first current backpressure coefficient 501. Calculator 414 (1_2) and Register 413 (1_2) are coupled to Register 412 (1_1) to receive a first-first previous backpressure coefficient 502 earlier than the first current backpressure coefficient 501. For example, the first current backpressure coefficient 501 may be the total delay time in the current time window, while the first-first previous backpressure coefficient 502 may be the total delay time in another time window preceding the current time window. Calculator 414 (1_2) is coupled to Register 413 (1_2) to receive a first-second previous backpressure coefficient 503 earlier than the first-first previous backpressure coefficient 502. Calculator 414 (1_2) uses the first current backpressure coefficient 501, the first-first previous backpressure coefficient 502, and the first-second previous backpressure coefficient 503 to calculate the backpressure coefficient BP4. The first access request path controller 420 is coupled to the first calculator 414 to receive the back pressure coefficient BP4.

[0079] The calculations of the first calculator 414 can be configured via registers for different calculations depending on the actual design and application. For example, the first calculator 414 uses the following formula 1 to calculate the back pressure coefficient. In formula 1, D represents the back pressure coefficient BP4, W1 represents the first weight, A represents the first current back pressure coefficient 501, W2 represents the second weight, B represents the first-first previous back pressure coefficient 502, W3 represents the third weight, and C represents the first-second previous back pressure coefficient 503. The first weight W1, the second weight W2, and the third weight W3 can be any real numbers determined according to the actual design and application. The first weight W1 is greater than the second weight W2, and the second weight W2 is greater than or equal to the third weight W3. For example, the first weight W1 is 3, the second weight W2 is 2, and the third weight W3 is 1. Or, the first weight W1 is 6, the second weight W2 is 2, and the third weight W3 is 1.

[0080] D = W1 A + W2 B + W3 C Calculation Formula 1

[0081] Figure 6 This is a circuit block diagram of a second artificial intelligence chip 600 according to another embodiment of the present invention. Figure 6 In the illustrated embodiment, the second artificial intelligence chip 600 includes multiple artificial intelligence (AI) cores (e.g., Figure 6 The number N of the AI ​​cores 610_1 to 610_N (shown as AI core 610_N, 2_1 to 2_N) can be determined according to the actual design and application. Figure 6The 2_1 AI core 610_1 to the 2_N AI core 610_N shown can be referenced. Figure 1 The relevant explanations of the first AI core 110_1 to the first AI core 110_N shown are as follows, and by analogy, they will not be repeated here.

[0082] Figure 1 The first artificial intelligence chip 100 shown also includes a task dispatch engine 601, a second flow control circuit 620, a second arbitrator 630, a second cache 640, and a second main memory 650. Figure 6 The second flow control circuit 620, the second arbitrator 630, the second cache 640, and the second main memory 650 shown can be referenced. Figure 1 The descriptions of the first flow control circuit 120, the first arbitrator 130, the first cache 140, and the first main memory 150 are shown below and can be extrapolated from there, so they will not be repeated. The task allocation engine 601 allocates one or more tasks to the second_1 AI cores 610_1 to the second_N AI cores 610_N. Furthermore, the task allocation engine 601 also notifies the second flow control circuit 620 of the task allocation information TD6.

[0083] Here, the 2_1 AI core 610_1 is used as an example. Other AI cores (such as the 2_N AI core 610_N) can be described by analogy to the relevant description of the 2_1 AI core 610_1, and therefore will not be elaborated upon. The second flow control circuit 620 counts the back pressure of the feedback data in the feedback data path from the second cache 640 through the second arbitrator 630 to the 2_1 AI core 610_1, in order to generate the first back pressure coefficient BP71 (not shown in the diagram). Figure 6 (To be explained later). The second flow control circuit 620 dynamically determines whether to adjust the first reverse voltage coefficient BP71 to generate the second reverse voltage coefficient BP72 based on the task allocation information TD6 (not shown in the diagram). Figure 6 (To be explained later). The second flow control circuit 620 dynamically determines whether to block the access request handshake between the second_1 AI core 610_1 and the second cache 640 based on the second backpressure coefficient BP72. Similarly, the second flow control circuit 620 counts the backpressure of the feedback data in the feedback data path from the second cache 640 through the second arbitrator 630 to the second_N AI core 610_N to generate the third backpressure coefficient BP73 (not shown in the diagram). Figure 6 (To be explained later). The second flow control circuit 620 dynamically determines whether to adjust the third reverse voltage coefficient BP73 to generate the fourth reverse voltage coefficient BP74 based on the task allocation information TD6 (not shown in the diagram). Figure 6(To be explained later). The second flow control circuit 620 dynamically determines whether to block the access request handshake between the AI ​​core 2_N artificial intelligence core 610_N and the second cache 640 based on the fourth reverse voltage coefficient BP74.

[0084] Figure 7 This is a circuit block diagram of the second flow control circuit 620, drawn according to an embodiment of the present invention. Figure 7 The second flow control circuit 620 shown can be used as Figure 6 This is one of many implementation examples of the second flow control circuit 620 shown. Figure 7 The task allocation engine 601, the second AI cores 610_1 to 2_N AI cores 610_N, the second flow control circuit 620, and the second arbitrator 630 shown can be referenced. Figure 6 The relevant explanations are omitted here. Figure 7 In the illustrated embodiment, the second flow control circuit 620 includes multiple flow controllers (e.g., Figure 7 The diagram shows the second-to-first flow controller 621_1, ..., the second-to-Nth flow controller 621_N, and the central flow controller 622. For ease of explanation, the second-to-first AI core 610_1 and the second-to-first flow controller 621_1 are used as examples below. The other AI cores (e.g., the second-to-Nth AI core 610_N) and the other flow controllers (e.g., the second-to-Nth flow controller 621_N) can be deduced by referring to the relevant descriptions of the second-to-first AI core 610_1 and the second-to-first flow controller 621_1, and therefore will not be elaborated upon.

[0085] The second-level flow controller 621_1 is coupled between the second-level AI core 610_1 and the second arbitrator 630. The second-level flow controller 621_1 calculates the backpressure of the feedback data in the feedback data path from the second cache 640 through the second arbitrator 630 to the second-level AI core 610_1 to generate a first backpressure coefficient BP71. The second-level flow controller 621_N calculates the backpressure of the feedback data in the feedback data path from the second cache 640 through the second arbitrator 630 to the second-level AI core 610_N to generate a third backpressure coefficient BP73. The central flow controller 622 is coupled to the second-level flow controller 621_1 to receive the first backpressure coefficient BP71. The central flow controller 622 is coupled to the second-level flow controller 621_N to receive the third backpressure coefficient BP73.

[0086] The task allocation engine also informs the central flow controller 622 of the task allocation information TD6. The central flow controller 622 dynamically decides whether to adjust the 1st backpressure coefficient BP71 to generate a 2nd backpressure coefficient BP72 to the 2nd_1 flow controller 621_1 according to the task allocation information TD6. Based on the task allocation information TD6, the central flow controller 622 can employ different allocation strategies for different configurations. For example, in the case that the 2nd_1 artificial intelligence core 610_1 to the 2nd_N artificial intelligence core 610_N run the same program, the central flow controller 622 will adjust the 1st backpressure coefficient BP71 based on the percentage of the task proportion to generate the 2nd backpressure coefficient BP72 to the 2nd_1 flow controller 621_1. If the 2nd_1 artificial intelligence core 610_1 to the 2nd_N artificial intelligence core 610_N run different programs, the central flow controller 622 will directly pass the 1st backpressure coefficient BP71 generated by the 2nd_1 flow controller 621_1 as the 2nd backpressure coefficient BP72 back to the 2nd_1 flow controller 621_1. The 2nd_1 flow controller 621_1 dynamically decides whether to block the access request handshake between the 2nd_1 artificial intelligence core 610_1 and the 2nd cache 640 according to the 2nd backpressure coefficient BP72. By analogy, the central flow controller 622 dynamically decides whether to adjust the 3rd backpressure coefficient BP73 to generate the 4th backpressure coefficient BP74 to the 2nd_N flow controller 621_N according to the task allocation information TD6. The 2nd_N flow controller 621_N dynamically decides whether to block the access request handshake between the AI core 2nd_N artificial intelligence core 610_N and the 2nd cache 640 according to the 4th backpressure coefficient BP74.

[0087] Figure 8 is shown in accordance with an embodiment of the present application, a circuit block schematic diagram of the 2nd_1 flow controller 621_1. Figure 8 The 2nd_1 flow controller 621_1 shown can be used as Figure 7 One of the many embodiments of the 2nd_1 flow controller 621_1 shown. Figure 8 The 2nd_1 artificial intelligence core 610_1, the 2nd_1 flow controller 621_1, the central flow controller 622, and the 2nd arbiter 630 shown can refer to the related description of Figure 7 the rest of the AI cores (such as the 2nd_N artificial intelligence core 610_N) and the rest of the flow controllers (such as the 2nd_N flow controller 621_N) can refer to the related description of the 2nd_1 artificial intelligence core 610_1 and the 2nd_1 flow controller 621_1 and be analogized, so it will not be described again. In the embodiment shown Figure 8 The 2nd_1 flow controller 621_1 includes a 2nd data path backpressure counting circuit 810 and a 2nd access request path controller 820.

[0088] The second data path backpressure counting circuit 810 is coupled between the second_1 artificial intelligence core 610_1 and the second arbiter 630. The second data path backpressure counting circuit 810 counts backpressure of feedback data in a feedback data path from the second cache 640 through the second arbiter 630 to the second_1 artificial intelligence core 610_1 to generate a first backpressure coefficient BP71. The second data path backpressure counting circuit 810 is coupled to the central flow controller 622 to provide the first backpressure coefficient BP71. The second access request path controller 820 is coupled between the second_1 artificial intelligence core 610_1 and the second arbiter 630. The second access request path controller 820 is coupled to the central flow controller 622 to receive a second backpressure coefficient BP72. The second access request path controller 820 dynamically decides whether to block an access request handshake between the second_1 artificial intelligence core 610_1 and the second cache 640 according to the second backpressure coefficient BP72.

[0089] Figure 9 FIG. 8 shows a circuit block diagram of the second data path backpressure counting circuit 810 according to an embodiment of the present application. Figure 9 The second data path backpressure counting circuit 810 can be implemented as Figure 8 The second data path backpressure counting circuit 810 can be implemented as Figure 9 The second_1 artificial intelligence core 610_1, the second data path backpressure counting circuit 810, the second access request path controller 820, and the second arbiter 630 can be implemented as Figure 8 The second_1 artificial intelligence core 610_1, the second data path backpressure counting circuit 810, the second access request path controller 820, and the second arbiter 630 can be implemented as Figure 9 In the embodiment shown in FIG. 8, the second data path backpressure counting circuit 810 includes a second data path backpressure counter 811, a second_1 register 812, a second_2 register 813, and a second calculator 814.

[0090] The second data path backpressure counter 811 is coupled between the second_1 artificial intelligence core 610_1 and the second arbiter 630. The second data path backpressure counter 811 counts backpressure of feedback data in a feedback data path from the second cache 640 through the second arbiter 630 to the second_1 artificial intelligence core 610_1 to generate a second current backpressure coefficient 901. Figure 9 The second data path backpressure counter 811 can be implemented as Figure 5 The second data path backpressure counter 811 can be implemented as

[0091] The 2_1 register 812 and the 2 calculator 814 are coupled to the 2 data path back pressure counter 811 to receive a 2 current back pressure coefficient 901. The 2_2 register 813 and the 2 calculator 814 are coupled to the 2_1 register 812 to receive a 2_1 previous back pressure coefficient 902 that is older than the 2 current back pressure coefficient 901. For example, the 2 current back pressure coefficient 901 can be a total delay time long in a current time window, and the 2_1 previous back pressure coefficient 902 can be a total delay time long in another time window that is before the current time window. The 2 calculator 814 is also coupled to the 2_2 register 813 to receive a 2_2 previous back pressure coefficient 903 that is older than the 2_1 previous back pressure coefficient 902. The 2 calculator 814 uses the 2 current back pressure coefficient 901, the 2_1 previous back pressure coefficient 902, and the 2_2 previous back pressure coefficient 903 to calculate the 1 back pressure coefficient BP71.

[0092] The 2 calculator 814 is coupled to the central flow controller 622 to provide the 1 back pressure coefficient BP71. Figure 9 The 2 calculator 814 can refer to the related description of the 1 calculator 414 shown above and be analogized. For example, the 2 calculator 814 uses the above calculation equation 1 to calculate the 1 back pressure coefficient BP71. In the calculation equation 1, D represents the 1 back pressure coefficient BP71, W1 represents a first weight, A represents the 2 current back pressure coefficient 901, W2 represents a second weight, B represents the 2_1 previous back pressure coefficient 902, W3 represents a third weight, and C represents the 2_2 previous back pressure coefficient 903. Figure 5 The 2 calculator 814 is coupled to the central flow controller 622 to provide the 1 back pressure coefficient BP71. Figure 9 The 2 calculator 814 can refer to the related description of the 1 calculator 414 shown above and be analogized. For example, the 2 calculator 814 uses the above calculation equation 1 to calculate the 1 back pressure coefficient BP71. In the calculation equation 1, D represents the 1 back pressure coefficient BP71, W1 represents a first weight, A represents the 2 current back pressure coefficient 901, W2 represents a second weight, B represents the 2_1 previous back pressure coefficient 902, W3 represents a third weight, and C represents the 2_2 previous back pressure coefficient 903.

Claims

1. An artificial intelligence chip, characterized by, The artificial intelligence chip comprises: a first artificial intelligence core; a cache; an arbiter coupled to the cache; and a flow control circuit coupled between the first artificial intelligence core and the arbiter, wherein the flow control circuit counts back pressure of feedback data in a first feedback data path from the cache to the first artificial intelligence core through the arbiter; and the flow control circuit dynamically decides whether to block an access request handshake between the first artificial intelligence core and the cache according to the back pressure of the feedback data. The artificial intelligence chip further comprises a second artificial intelligence core, the arbiter is coupled between the second artificial intelligence core and the cache, the flow control circuit is coupled between the second artificial intelligence core and the arbiter, the flow control circuit counts back pressure of feedback data in a second feedback data path from the cache to the second artificial intelligence core through the arbiter, and the flow control circuit dynamically decides whether to block an access request handshake between the second artificial intelligence core and the cache according to the back pressure of the feedback data in the second feedback data path.

2. The artificial intelligence chip according to claim 1, wherein, The flow control circuit comprises: a first flow controller coupled between the first artificial intelligence core and the arbiter, wherein the first flow controller counts the back pressure of the feedback data in the first feedback data path from the cache to the first artificial intelligence core through the arbiter, and the first flow controller dynamically decides whether to block the access request handshake between the first artificial intelligence core and the cache according to the back pressure of the feedback data in the first feedback data path; and a second flow controller coupled between the second artificial intelligence core and the arbiter, wherein the second flow controller counts the back pressure of the feedback data in the second feedback data path from the cache to the second artificial intelligence core through the arbiter, and the second flow controller dynamically decides whether to block the access request handshake between the second artificial intelligence core and the cache according to the back pressure of the feedback data in the second feedback data path.

3. The artificial intelligence chip of claim 2, wherein, The first flow controller comprises: a data path back pressure counting circuit coupled between the first artificial intelligence core and the arbiter, wherein the data path back pressure counting circuit counts the back pressure of the feedback data in the first feedback data path from the cache to the first artificial intelligence core through the arbiter to generate a back pressure coefficient; and an access request path controller coupled between the first artificial intelligence core and the arbiter, wherein the access request path controller is coupled to the data path back pressure counting circuit to receive the back pressure coefficient, and the access request path controller dynamically decides whether to block the access request handshake between the first artificial intelligence core and the cache according to the back pressure coefficient.

4. The artificial intelligence chip according to claim 3, characterized in that, The data path back pressure counting circuit comprises: a data path backpressure counter coupled between the first artificial intelligence core and the arbiter, wherein the data path backpressure counter counts the backpressure condition of the feedback data in the first feedback data path from the cache through the arbiter to the first artificial intelligence core to generate a current backpressure coefficient; a first register coupled to the data path backpressure counter to receive the current backpressure coefficient; and a calculator coupled to the data path backpressure counter to receive the current backpressure coefficient and coupled to the first register to receive a first previous backpressure coefficient earlier than the current backpressure coefficient, wherein the calculator calculates the backpressure coefficient using the current backpressure coefficient and the first previous backpressure coefficient, and the access request path controller is coupled to the calculator to receive the backpressure coefficient.

5. The artificial intelligence chip according to claim 4, characterized in that, the data path backpressure counter circuit further comprises: a second register coupled to the first register to receive the first previous backpressure coefficient, wherein the calculator is further coupled to the second register to receive a second previous backpressure coefficient earlier than the first previous backpressure coefficient, and the calculator calculates the backpressure coefficient using the current backpressure coefficient, the first previous backpressure coefficient, and the second previous backpressure coefficient.

6. The artificial intelligence chip according to claim 5, characterized in that, The calculator calculates the back pressure coefficient using a calculation D = W1 A + W2 B + W3 C where D represents the back pressure coefficient, W1 represents a first weight, A represents the current back pressure coefficient, W2 represents a second weight, B represents the first previous back pressure coefficient, W3 represents a third weight, and C represents the second previous back pressure coefficient.

7. The artificial intelligence chip according to claim 6, characterized in that, the first weight W1 is greater than the second weight W2, and the second weight W2 is greater than or equal to the third weight W3.

8. The artificial intelligence chip of claim 1, wherein, the flow control circuit counts the backpressure condition of the feedback data in the first feedback data path from the cache through the arbiter to the first artificial intelligence core to generate a first backpressure coefficient, the artificial intelligence chip further comprises a task assignment engine, the task assignment engine assigns a task to the first artificial intelligence core, the task assignment engine further informs a task assignment information to the flow control circuit, the flow control circuit dynamically decides whether to adjust the first backpressure coefficient to generate a second backpressure coefficient according to the task assignment information, and the flow control circuit dynamically decides whether to block the access request handshake between the first artificial intelligence core and the cache according to the second backpressure coefficient.

9. The artificial intelligence chip according to claim 8, characterized in that, the task assignment engine assigns a task to the second artificial intelligence core, the flow control circuit counts the backpressure condition of the feedback data in a second feedback data path from the cache through the arbiter to the second artificial intelligence core to generate a third backpressure coefficient, the flow control circuit dynamically decides whether to adjust the third backpressure coefficient to generate a fourth backpressure coefficient according to the task assignment information, and the flow control circuit dynamically decides whether to block the access request handshake between the second artificial intelligence core and the cache according to the fourth backpressure coefficient.

10. The artificial intelligence chip according to claim 9, characterized in that, the flow control circuit comprises: a first flow controller coupled between the first artificial intelligence core and the arbiter, wherein the first flow controller counts the backpressure condition of the feedback data in the first feedback data path from the cache through the arbiter to the first artificial intelligence core to generate the first backpressure coefficient; a second flow controller coupled between the second artificial intelligence core and the arbiter, wherein the second flow controller counts the backpressure condition of the feedback data in the second feedback data path from the cache through the arbiter to the second artificial intelligence core to generate the third backpressure coefficient; and a central flow controller coupled to the first flow controller to receive the first backpressure coefficient and coupled to the second flow controller to receive the third backpressure coefficient, wherein the task allocation engine further informs task allocation information to the central flow controller, the central flow controller dynamically decides whether to adjust the first backpressure coefficient to generate the second backpressure coefficient to the first flow controller according to the task allocation information, the central flow controller dynamically decides whether to adjust the third backpressure coefficient to generate the fourth backpressure coefficient to the second flow controller according to the task allocation information, the first flow controller dynamically decides whether to block the access request handshake between the first artificial intelligence core and the cache according to the second backpressure coefficient, and the second flow controller dynamically decides whether to block the access request handshake between the second artificial intelligence core and the cache according to the fourth backpressure coefficient.

11. The artificial intelligence chip according to claim 10, characterized in that, the first flow controller comprises: a data path backpressure counting circuit coupled between the first artificial intelligence core and the arbiter, wherein the data path backpressure counting circuit counts the backpressure condition of the feedback data in the first feedback data path from the cache through the arbiter to the first artificial intelligence core to generate the first backpressure coefficient, and the data path backpressure counting circuit is coupled to the central flow controller to provide the first backpressure coefficient; and an access request path controller coupled between the first artificial intelligence core and the arbiter, wherein the access request path controller is coupled to the central flow controller to receive the second backpressure coefficient, and the access request path controller dynamically decides whether to block the access request handshake between the first artificial intelligence core and the cache according to the second backpressure coefficient.

12. The artificial intelligence chip according to claim 11, characterized in that, the data path backpressure counting circuit comprises: a data path backpressure counter coupled between the first artificial intelligence core and the arbiter, wherein the data path backpressure counter counts the backpressure condition of the feedback data in the first feedback data path from the cache through the arbiter to the first artificial intelligence core to generate a current backpressure coefficient; a first register coupled to the data path backpressure counter to receive the current backpressure coefficient; and a calculator coupled to the data path backpressure counter to receive the current backpressure coefficient and coupled to the first register to receive a first previous backpressure coefficient earlier than the current backpressure coefficient, wherein the calculator uses the current backpressure coefficient and the first previous backpressure coefficient to calculate the first backpressure coefficient, and the calculator is coupled to the central flow controller to provide the first backpressure coefficient.

13. The artificial intelligence chip according to claim 12, characterized in that, The data path backpressure counting circuit further includes: a second register coupled to the first register to receive the first previous backpressure coefficient, wherein the calculator is further coupled to the second register to receive a second previous backpressure coefficient earlier than the first previous backpressure coefficient, and the calculator uses the current backpressure coefficient, the first previous backpressure coefficient, and the second previous backpressure coefficient to calculate the first backpressure coefficient.

14. The artificial intelligence chip according to claim 13, characterized in that, The calculator calculates the first back pressure coefficient using a calculation D = W1 A + W2 B + W3 C where D represents the first back pressure coefficient, W1 represents a first weight, A represents the current back pressure coefficient, W2 represents a second weight, B represents the first previous back pressure coefficient, W3 represents a third weight, and C represents the second previous back pressure coefficient.

15. The artificial intelligence chip according to claim 14, characterized in that, The first weight W1 is greater than the second weight W2, and the second weight W2 is greater than or equal to the third weight W3.

16. A method of operating an artificial intelligence chip, comprising: The operation method includes: counting, by a flow control circuit of the artificial intelligence chip, backpressure of feedback data in a first feedback data path from a cache of the artificial intelligence chip through an arbiter of the artificial intelligence chip to a first artificial intelligence core of the artificial intelligence chip, wherein the arbiter is coupled to the cache, and the flow control circuit is coupled between the first artificial intelligence core and the arbiter; and dynamically deciding, by the flow control circuit, whether to block a handshake of an access request between the first artificial intelligence core and the cache according to the backpressure of the feedback data. The artificial intelligence chip further includes a second artificial intelligence core, the arbiter is coupled between the second artificial intelligence core and the cache, the flow control circuit is coupled between the second artificial intelligence core and the arbiter, and the operation method further includes: counting, by the flow control circuit, backpressure of feedback data in a second feedback data path from the cache through the arbiter to the second artificial intelligence core; and dynamically deciding, by the flow control circuit, whether to block a handshake of an access request between the second artificial intelligence core and the cache according to the backpressure of the feedback data in the second feedback data path.

17. The method of operation of claim 16, wherein, The operation method further includes: counting, by a first flow controller of the flow control circuit, the backpressure of the feedback data in the first feedback data path from the cache through the arbiter to the first artificial intelligence core, wherein the first flow controller is coupled between the first artificial intelligence core and the arbiter; dynamically deciding, by the first flow controller, whether to block the handshake of the access request between the first artificial intelligence core and the cache according to the backpressure of the feedback data in the first feedback data path; counting, by a second flow controller of the flow control circuit, the backpressure of the feedback data in the second feedback data path from the cache through the arbiter to the second artificial intelligence core, wherein the second flow controller is coupled between the second artificial intelligence core and the arbiter; and dynamically deciding, by the second flow controller, whether to block the handshake of the access request between the second artificial intelligence core and the cache according to the backpressure of the feedback data in the second feedback data path.

18. The method of claim 16, wherein, The artificial intelligence chip further includes a task allocation engine, and the operation method further includes: statistically count, by the flow control circuit, the backpressure condition of the feedback data in the first feedback data path from the cache to the first artificial intelligence core through the arbiter to generate a first backpressure coefficient; allocate, by the task allocation engine, a task to the first artificial intelligence core; notify, by the task allocation engine, task allocation information to the flow control circuit; dynamically decide, by the flow control circuit, whether to adjust the first backpressure coefficient to generate a second backpressure coefficient according to the task allocation information; and dynamically decide, by the flow control circuit, whether to block the access request handshake between the first artificial intelligence core and the cache according to the second backpressure coefficient.

Images