Kernel architecture and scheduling methods that support concurrent execution of processing units and multiply-accumulate units.
By using a kernel architecture and scheduling method that enables concurrent execution, the problem of concurrent execution between thread blocks is solved, the time overlap between matrix calculation and post-processing stages is achieved, the utilization rate and computational efficiency of execution units are improved, and the risks of resource contention and data overwriting are reduced.
Patent Information
- Application Number
- CN202611568387.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-12
- Publication Date
- 2026-11-10
AI Technical Summary
When performing neural network calculations, the existing kernel architecture has difficulty ensuring the safe concurrency of matrix calculation and post-processing stages in different thread blocks while maintaining the correspondence between thread block instructions and data. This can easily lead to instruction mismatch, execution resource contention, register data overwriting, or memory access conflicts.
A kernel architecture and scheduling method that supports concurrent execution of processing units and multiply-accumulate units are adopted. Through hardware scheduling and division of execution resources, the scheduler, instruction decoding circuit, thread block selection circuit and concurrent dispatch circuit are used to establish queue access, request passing, decoding transmission and dispatch control paths to ensure that the post-processing stage of the current thread block and the matrix calculation stage of the next thread block overlap in time.
It reduces waiting caused by stage serial connection, improves the utilization of execution units, reduces the complexity of manually orchestrating cross-thread block pipelines, avoids resource contention and data overwriting, and improves computational efficiency.
Smart Images

Figure CN123068356A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of integrated circuits and artificial intelligence processors, and in particular to a kernel architecture and scheduling method that supports concurrent execution of processing units and multiply-accumulate units. Background Technology
[0002] When an artificial intelligence processor performs neural network calculations, it typically needs to sequentially complete matrix multiplication, dequantization, accumulation, normalization, positional encoding, and nonlinear operations. Different computational stages have different hardware resource requirements, and the post-processing stage after matrix calculations may leave some computing resources in a waiting state.
[0003] Current kernels typically complete all operations one thread block at a time before starting the next. Directly overlapping operations between different thread blocks can easily lead to instruction mismatches, resource contention, register data overwriting, or memory access conflicts. Therefore, ensuring the safe concurrency of matrix computation and post-processing stages in different thread blocks while maintaining the correct instruction-data mapping between thread blocks is a pressing technical problem that needs to be solved. Summary of the Invention
[0004] This application proposes a kernel architecture and scheduling method that supports concurrent execution of processing units and multiply-accumulate units, aiming to make the post-processing stage of the current thread block and the matrix calculation stage of the next thread block overlap in time through hardware scheduling and division of execution resources.
[0005] In a first aspect, embodiments of this application provide a kernel architecture that supports concurrent execution of processing units and multiply-accumulate units, including: The scheduler includes a command FIFO buffer circuit, an instruction decoding circuit, a thread block selection circuit, and a concurrent dispatch circuit; the queue interaction interface of the command FIFO buffer circuit is electrically connected to the queue access interface of the thread block selection circuit; the scheduling request interface of the instruction decoding circuit is electrically connected to the request receiving interface of the thread block selection circuit; the decoding sending interface of the instruction decoding circuit is electrically connected to the decoding receiving interface of the concurrent dispatch circuit; and the dispatch control interface of the thread block selection circuit is electrically connected to the scheduling control interface of the concurrent dispatch circuit. The multiply-accumulate unit has its matrix instruction receiving interface electrically connected to the first instruction distribution interface of the concurrent distribution circuit; At least two processing units, including a first processing unit and a second processing unit; the auxiliary instruction receiving interface of the first processing unit is electrically connected to the second instruction distribution interface of the concurrent distribution circuit; the post-processing instruction receiving interface of the second processing unit is electrically connected to the third instruction distribution interface of the concurrent distribution circuit. The instruction decoding circuit is used to parse the concurrent request instruction of the current thread block after the matrix calculation stage is completed; the thread block selection circuit is used to check the queue status in response to the parsing result; when the queue is not empty, the thread block selection circuit reads the command information; the thread block selection circuit determines the next thread block according to the command information; the concurrent dispatch circuit is used to dispatch the matrix multiplication instruction of the next thread block to the multiply-accumulate unit through the first instruction dispatch interface; dispatch the matrix auxiliary operation instruction of the next thread block to the first processing unit through the second instruction dispatch interface; dispatch the post-processing instruction of the current thread block to the second processing unit through the third instruction dispatch interface, so that the matrix calculation stage of the next thread block and the post-processing stage of the current thread block overlap in execution time; the multiply-accumulate unit performs matrix multiplication operation; the first processing unit performs matrix auxiliary operation; the second processing unit performs post-processing operation; the thread block is a group of threads that execute the same kernel function; the current thread block is the thread block to which the concurrent request instruction belongs; the next thread block is a thread block to be scheduled that is different from the current thread block; the matrix calculation stage includes matrix multiplication operation and matrix auxiliary operation.
[0006] As can be seen, this embodiment establishes queue access, request transmission, decoding transmission, and distribution control paths through interconnected interfaces, and establishes instruction distribution paths leading to the multiply-accumulate unit, the first processing unit, and the second processing unit, respectively. The thread block selection circuit responds to concurrent requests to determine the next thread block, and the concurrent distribution circuit sends the matrix multiplication instructions and matrix auxiliary operation instructions of the next thread block to the multiply-accumulate unit and the first processing unit, while sending the post-processing instructions of the current thread block to the second processing unit. Therefore, matrix auxiliary operations and post-processing operations can occupy different processing resources, which helps to reduce the waiting caused by stage serial connection, improve the utilization rate of execution units, and reduce the complexity of manually orchestrating cross-thread block pipelines.
[0007] In one possible example, the command FIFO cache circuit includes a command storage array, a read pointer register, a write pointer register, a queue state generation circuit, and a read / write control circuit. The command receiving interface of the read / write control circuit is electrically connected to the command sending interface of the upper-level scheduling circuit, and is used to receive command information and write control signals from the thread block to be scheduled. The queue interaction interface of the read / write control circuit constitutes the queue interaction interface of the command FIFO cache circuit, and is used to transmit read control signals, queue state, and command information with the queue access interface. The storage access interface of the read / write control circuit is electrically connected to the storage read / write interface of the command storage array. The read pointer access interface of the read / write control circuit is electrically connected to the read pointer read / write interface of the read pointer register. The write pointer access interface of the read / write control circuit is electrically connected to the write pointer read / write interface of the write pointer register. The write interface is electrically connected; the read pointer sending interface of the read pointer register is electrically connected to the first pointer receiving interface of the queue state generation circuit; the write pointer sending interface of the write pointer register is electrically connected to the second pointer receiving interface of the queue state generation circuit; the queue state sending interface of the queue state generation circuit is electrically connected to the queue state receiving interface of the read / write control circuit; the read / write control circuit is used to write the command information into the command storage array according to the write pointer; after writing is completed, the read / write control circuit updates the write pointer; the read / write control circuit is also used to respond to the read control signal and read the earliest written and yet-to-be-read command information according to the read pointer; after the command information is received by the thread block selection circuit, the read / write control circuit updates the read pointer; the queue state generation circuit is used to generate the queue state according to the read pointer and the write pointer.
[0008] As can be seen, this example coordinates the command storage array and read / write pointers through the read / write control circuit, so that the command information of the thread block to be scheduled is read in the order of writing, and provides the queue status corresponding to the command information to the thread block selection circuit, which helps to avoid duplicate scheduling, missed scheduling, and accidental reading of empty queues.
[0009] In one possible example, the scheduler further includes an instruction fetch circuit; the fetch control interface of the thread block selection circuit is electrically connected to the thread selection interface of the instruction fetch circuit; the instruction storage access interface of the instruction fetch circuit is electrically connected to the instruction storage service interface of the instruction memory; the instruction providing interface of the instruction fetch circuit is electrically connected to the instruction fetch interface of the instruction decoding circuit; the thread block selection circuit is used to provide thread block selection information via the fetch control interface; the instruction fetch circuit is used to fetch the instruction of the corresponding thread block according to the thread block selection information; the instruction fetch circuit provides the instruction to the instruction decoding circuit through the instruction providing interface; the thread block selection information is used to select the fetch object between the current thread block and the next thread block; The instruction decoding circuit provides decoding instructions carrying thread block identifiers and execution unit identifiers to the decoding sending interface; the concurrent distribution circuit receives the decoding instructions through the decoding receiving interface; the concurrent distribution circuit selects an instruction path based on the execution unit identifier; when the execution unit identifier indicates the multiply-accumulate unit, the instruction received by the decoding receiving interface is routed to the first instruction distribution interface; when the execution unit identifier indicates the first processing unit, the instruction received by the decoding receiving interface is routed to the second instruction distribution interface; when the execution unit identifier indicates the second processing unit, the instruction received by the decoding receiving interface is routed to the third instruction distribution interface; the concurrent distribution circuit retains the corresponding thread block identifier during instruction distribution.
[0010] As can be seen, this example determines the instruction fetch object through thread block selection information and retains the thread block identifier and execution unit identifier in the decoding instruction, so that the same decoding input can be accurately routed to different execution units, which helps to prevent the instructions of the current thread block from being confused with those of the next thread block.
[0011] In one possible example, the thread block selection circuit includes a selection control circuit and a scheduling status register; the status access interface of the selection control circuit is electrically connected to the scheduling status read / write interface of the scheduling status register; the queue access interface of the selection control circuit constitutes the queue access interface of the thread block selection circuit; the request receiving interface of the selection control circuit constitutes the request receiving interface of the thread block selection circuit; the distribution control interface of the selection control circuit constitutes the distribution control interface of the thread block selection circuit; the first execution status sending interface of the multiply-accumulate unit is electrically connected to the first execution status receiving interface of the selection control circuit; the second execution status sending interface of the first processing unit is electrically connected to the second execution status receiving interface of the selection control circuit; and the third execution status sending interface of the second processing unit is electrically connected to the third execution status receiving interface of the selection control circuit. The selection control circuit updates the scheduling status register based on the received execution status. The scheduling status register records the matrix calculation completion status, post-processing completion status, and occupancy status of the second processing unit for the thread block. After the matrix calculation phase of the next thread block is completed, the selection control circuit checks the occupancy status. If the second processing unit is occupied, the selection control circuit records the post-processing pending launch status of the next thread block. If the second processing unit is idle, the selection control circuit sends a post-processing distribution permission via the distribution control interface. The concurrent distribution circuit receives the post-processing distribution permission through the scheduling control interface and distributes the post-processing instructions for the next thread block. The selection control circuit also checks the queue status when responding to the concurrent request instruction. When the queue is empty, a new thread block is not started, and the current thread block is allowed to continue executing the post-processing phase.
[0012] As can be seen, this example uses the scheduling status register to store the matrix calculation completion status, post-processing completion status, and processing unit occupancy status, enabling the selection control circuit to control the issuance of post-processing instructions for different completion orders of the two thread blocks. This helps to avoid repeated issuance of post-processing instructions or premature issuance when the second processing unit is occupied.
[0013] In one possible example, the multiply-accumulate unit includes a multiply-accumulate control circuit and a multiply-accumulate array; the matrix instruction receiving interface of the multiply-accumulate control circuit constitutes the matrix instruction receiving interface of the multiply-accumulate unit; the multiply-accumulate control sending interface of the multiply-accumulate control circuit is electrically connected to the multiply-accumulate control receiving interface of the multiply-accumulate array; the multiply-accumulate control circuit is used to generate a multiply-accumulate control signal according to a matrix multiplication instruction; the multiply-accumulate array is used to perform matrix multiplication operations in response to the multiply-accumulate control signal; the first processing unit includes a first arithmetic circuit; the auxiliary instruction receiving interface of the first arithmetic circuit constitutes the auxiliary instruction receiving interface; the first arithmetic circuit is dedicated to performing multiplication operations with... The second processing unit includes at least one of matrix multiplication-related inverse quantization, post-summation, and accumulation operations; the second processing unit includes a second arithmetic circuit; the post-processing instruction receiving interface of the second arithmetic circuit constitutes the post-processing instruction receiving interface; the second arithmetic circuit is used to execute post-processing instructions or general calculation instructions, the post-processing instructions or general calculation instructions include at least one of root mean square normalization instructions, position encoding instructions, and nonlinear operation instructions; the second arithmetic circuit and the first arithmetic circuit are different hardware arithmetic circuits; when the second arithmetic circuit executes the post-processing instructions of the current thread block, it does not occupy the first arithmetic circuit and the multiply-accumulate array.
[0014] As can be seen, this example configures matrix multiplication, matrix auxiliary operations, and post-processing operations to different hardware operation circuits, so that the post-processing of the current thread block will not occupy the first operation circuit and multiply-accumulate array required by the next thread block, thus providing conditions for concurrent execution across thread blocks from the hardware resource level.
[0015] In one possible example, the kernel architecture further includes a first register file, a second register file, and a third register file that are physically independent of each other; the first register access interface of the multiply-accumulate unit is electrically connected to the first storage read / write interface of the first register file; the second register access interface of the first processing unit is electrically connected to the second storage read / write interface of the second register file; the third register access interface of the second processing unit is electrically connected to the third storage read / write interface of the third register file; during the concurrent execution of the current thread block and the next thread block, the multiply-accumulate unit accesses the matrix multiplication operation data of the next thread block via the first register access interface; the first processing unit accesses the matrix auxiliary operation data of the next thread block via the second register access interface; and the second processing unit accesses the post-processing operation data of the current thread block via the third register access interface.
[0016] As can be seen, this example uses independent register files to isolate the computational data of different execution units and different thread blocks, which helps to reduce data overwriting and port contention caused by concurrent access, and enables each execution unit to independently carry out its corresponding computation.
[0017] In one possible example, the kernel architecture further includes a register storage array; the fourth register access interface of the multiply-accumulate unit is electrically connected to the fourth storage read / write interface of the register storage array; the fifth register access interface of the first processing unit is electrically connected to the fifth storage read / write interface of the register storage array; the sixth register access interface of the second processing unit is electrically connected to the sixth storage read / write interface of the register storage array; the register storage array has a first address range, a second address range, and a third address range that do not overlap; the fourth storage read / write interface is used to access the first address range; the fifth storage read / write interface is used to access the second address range; the sixth storage read / write interface is used to access the third address range; during the concurrent execution of the current thread block and the next thread block, the multiply-accumulate unit accesses the matrix multiplication operation data of the next thread block via the fourth register access interface; the first processing unit accesses the matrix auxiliary operation data of the next thread block via the fifth register access interface; the second processing unit accesses the post-processing operation data of the current thread block via the sixth register access interface.
[0018] As can be seen, this example achieves logical isolation of data processed by different threads by dividing non-overlapping address ranges within the same register storage array. While maintaining data isolation, it allows for sharing the physical resources of the register storage array, facilitating configuration between storage capacity and hardware area.
[0019] In one possible example, the kernel architecture further includes a memory access issue control circuit; the first memory access control interface of the multiply-accumulate unit is electrically connected to the first arbitration interface of the memory access issue control circuit; the second memory access control interface of the first processing unit is electrically connected to the second arbitration interface of the memory access issue control circuit; the third memory access control interface of the second processing unit is electrically connected to the third arbitration interface of the memory access issue control circuit; the first memory access control interface and the first arbitration interface are used to transmit the memory access request and corresponding authorization signal of the multiply-accumulate unit; the second memory access control interface and the second arbitration interface are used to transmit the memory access request and corresponding authorization signal of the first processing unit; the third memory access control interface and the third arbitration interface are used to transmit the memory access request and corresponding authorization signal of the second processing unit. The memory access launch control circuit is used to perform column-level launch arbitration. During the column-level launch cycle in which the multiply-accumulate unit is authorized, the memory access launch control circuit prohibits the first processing unit from issuing load or store instructions. The memory access launch control circuit also prohibits the second processing unit from issuing load or store instructions during this cycle. During any column-level launch cycle in which the processing unit is authorized, the memory access launch control circuit prohibits the multiply-accumulate unit from issuing load or store instructions. A single multiply-accumulate operation of the multiply-accumulate unit lasts for at least two column-level launch cycles. During the column-level launch cycle in which the multiply-accumulate unit does not issue load or store instructions, the memory access launch control circuit provides an authorization signal to the first processing unit or the second processing unit that has a memory access request.
[0020] As can be seen, this example refines memory access authorization to the column-level issue cycle. While preventing memory access conflicts between the multiply-accumulate unit and the processing unit in the same column-level issue cycle, it can utilize the cycle during which no memory access instruction is issued during the multiply-accumulate operation to process the memory access request of the processing unit, thereby improving the effective utilization of the memory access path.
[0021] Secondly, embodiments of this application provide a kernel scheduling method, applied to the kernel architecture described in any example of the first aspect. The method includes: the multiply-accumulate unit performing matrix multiplication of the current thread block; the first processing unit performing matrix auxiliary operations of the current thread block; after the matrix calculation of the current thread block is completed, the instruction decoding circuit parses the concurrent request instruction of the current thread block; the instruction decoding circuit transmits the parsing result to the request receiving interface via the scheduling request interface; the thread block selection circuit responds to the parsing result, accesses the queue interaction interface via the queue access interface, and obtains the queue status; when the queue is not empty, the thread block selection circuit reads the command information of the thread block to be scheduled; the thread block selection circuit determines the next thread block according to the command information; and the thread block selection circuit provides the scheduling control interface with the dispatch control interface. Thread block scheduling information; the instruction decoding circuit provides decoding instructions to the decoding receiving interface via the decoding sending interface; the concurrent distribution circuit, based on the thread block scheduling information, distributes the matrix multiplication instruction of the next thread block to the matrix instruction receiving interface via the first instruction distribution interface; distributes the matrix auxiliary operation instruction of the next thread block to the auxiliary instruction receiving interface via the second instruction distribution interface; distributes the post-processing instruction of the current thread block to the post-processing instruction receiving interface via the third instruction distribution interface; the multiply-accumulate unit performs the matrix multiplication operation of the next thread block; the first processing unit performs the matrix auxiliary operation of the next thread block; the second processing unit performs the post-processing operation of the current thread block; the scheduler arranges the instruction distribution timing so that the matrix calculation stage of the next thread block and the post-processing stage of the current thread block overlap in execution time.
[0022] As can be seen, this embodiment establishes queue access, request transmission, decoding transmission, and distribution control paths through interconnected interfaces, and establishes instruction distribution paths leading to the multiply-accumulate unit, the first processing unit, and the second processing unit, respectively. The thread block selection circuit responds to concurrent requests to determine the next thread block, and the concurrent distribution circuit sends the matrix multiplication instructions and matrix auxiliary operation instructions of the next thread block to the multiply-accumulate unit and the first processing unit, while sending the post-processing instructions of the current thread block to the second processing unit. Therefore, matrix auxiliary operations and post-processing operations can occupy different processing resources, which helps to reduce the waiting caused by stage serial connection, improve the utilization rate of execution units, and reduce the complexity of manually orchestrating cross-thread block pipelines.
[0023] In one possible example, the kernel architecture includes a selection control circuit and a scheduling status register. The selection control circuit updates the matrix computation completion status in the scheduling status register based on the execution status received by the first execution status receiving interface and the second execution status receiving interface. The selection control circuit updates the post-processing completion status and the occupancy status of the second processing unit based on the execution status received by the third execution status receiving interface. When the post-processing stage of the current thread block completes first, the selection control circuit updates the occupancy status of the second processing unit to idle. The multiply-accumulate unit continues to execute the matrix multiplication operation that the next thread block has not yet completed. The first processing unit continues to execute the matrix multiplication operation that the next thread block has not yet completed. The matrix-assisted operation is performed; when the matrix calculation stage of the next thread block is completed first, the selection control circuit checks the occupancy status of the second processing unit; if the second processing unit is still occupied by the current thread block, the selection control circuit records the post-processing pending launch status of the next thread block; after the matrix calculation stage of the next thread block is completed, when the second processing unit is idle, the selection control circuit sends a post-processing distribution permission to the scheduling control interface through the distribution control interface; the concurrent distribution circuit responds to the post-processing distribution permission and distributes the post-processing instruction of the next thread block to the post-processing instruction receiving interface through the third instruction distribution interface; the second processing unit executes the post-processing operation of the next thread block.
[0024] As can be seen, this example can handle two timing sequences: the current thread block completes its post-processing first, and the next thread block completes its matrix calculation first. By connecting the execution processes of two thread blocks in an occupied state and a post-processing pending state, it avoids issuing the post-processing instruction of the next thread block while the second processing unit is still occupied, and ensures that the post-processing process continues promptly after the second processing unit becomes idle.
[0025] As can be seen, in this embodiment, compared to the kernel architecture where the matrix computation stage and the post-processing stage are executed serially, this application achieves execution overlap between different thread blocks through hardware scheduling. This application establishes queue access, request passing, decoding transmission, and distribution control paths through interconnected interfaces, and establishes instruction distribution paths leading to the multiply-accumulate unit, the first processing unit, and the second processing unit, respectively. Since the thread block selection circuit can respond to concurrent requests to determine the next thread block, the concurrent distribution circuit can send the matrix multiplication instructions and matrix auxiliary operation instructions of the next thread block to the multiply-accumulate unit and the first processing unit, while simultaneously sending the post-processing instructions of the current thread block to the second processing unit. This allows matrix auxiliary operations and post-processing operations to occupy different processing resources, which helps reduce waiting caused by serial stage connections, improves the utilization rate of execution units, and reduces the complexity of manually orchestrating cross-thread block pipelines. Attached Figure Description
[0026] Figure 1 This is a schematic diagram of the overall structure of the kernel architecture that supports concurrent execution of the processing unit and the multiply-accumulate unit, as provided in the embodiments of this application.
[0027] Figure 2 A schematic diagram of the command first-in-first-out (FIFO) cache circuit provided in the embodiments of this application.
[0028] Figure 3 This is a schematic diagram of the instruction acquisition and instruction routing structure provided in the embodiments of this application.
[0029] Figure 4 This is a schematic diagram of the thread block selection circuit and execution status transmission structure provided in the embodiments of this application.
[0030] Figure 5 This is a schematic diagram of the internal structure of the multiply-accumulate unit, the first processing unit, and the second processing unit provided in the embodiments of this application.
[0031] Figure 6 This is a schematic diagram of the physical isolation register structure provided in an embodiment of this application.
[0032] Figure 7 This is a schematic diagram of the logical isolation register structure provided in an embodiment of this application.
[0033] Figure 8 This is a schematic diagram of the memory access and launch control structure provided in an embodiment of this application.
[0034] Figure 9 This is a schematic diagram of the kernel scheduling method provided in an embodiment of this application.
[0035] Figure 10 This is a schematic diagram of a method for post-processing scheduling based on the execution completion order provided in an embodiment of this application.
[0036] [Explanation of Labels in the Attached Image] 100. Kernel Architecture; 110. Scheduler; 111. Command FIFO Cache Circuit; 1111. Command Memory Array; 1112. Read Pointer Register; 1113. Write Pointer Register; 1114. Queue State Generation Circuit; 1115. Read / Write Control Circuit; 112. Instruction Decoding Circuit; 113. Thread Block Selection Circuit; 1131. Selection Control Circuit; 1132. Scheduler Status Register; 114. Concurrency Dispatch Circuit; 115. Instruction Fetch Circuit; 116. Instruction Storage Memory; 120, Multiply-accumulate unit; 121, Multiply-accumulate control circuit; 122, Multiply-accumulate array; 130, First processing unit; 131, First arithmetic circuit; 140, Second processing unit; 141, Second arithmetic circuit; 150, First register file; 160, Second register file; 170, Third register file; 180, Register storage array; 1801, First address range; 1802, Second address range; 1803, Third address range; 190, Memory access issue control circuit; 2101. Queue Interaction Interface; 2102. Queue Access Interface; 2103. Scheduling Request Interface; 2104. Request Receive Interface; 2105. Decode Send Interface; 2106. Decode Receive Interface; 2107. Distribution Control Interface; 2108. Scheduling Control Interface; 2109. Matrix Instruction Receive Interface; 2110. First Instruction Distribution Interface; 2111. Auxiliary Instruction Receive Interface; 2112. Second Instruction Distribution Interface; 2113. Post-processing Instruction Receive Interface; 2114. Third Instruction Distribution Interface; 2121. Command receiving interface; 2122. Command sending interface; 2123. Storage access interface; 2124. Storage read / write interface; 2125. Read pointer access interface; 2126. Read pointer read / write interface; 2127. Write pointer access interface; 2128. Write pointer read / write interface; 2129. Read pointer sending interface; 2130. First pointer receiving interface; 2131. Write pointer sending interface; 2132. Second pointer receiving interface; 2133. Queue status sending interface; 2134. Queue status receiving interface; 2141. Instruction Fetch Control Interface; 2142. Thread Selection Interface; 2143. Instruction Store Access Interface; 2144. Instruction Store Service Interface; 2145. Instruction Provider Interface; 2146. Instruction Fetch Interface; 2151. Status Access Interface; 2152. Scheduling Status Read / Write Interface; 2153. First Execution Status Send Interface; 2154. First Execution Status Receive Interface; 2155. Second Execution Status Send Interface; 2156. Second Execution Status Receive Interface; 2157. Third Execution Status Send Interface; 2158. Third Execution Status Receive Interface; 2161. Multiply-accumulate control transmit interface; 2162. Multiply-accumulate control receive interface; 2171. First register access interface; 2172. First memory read / write interface; 2173. Second register access interface; 2174. Second memory read / write interface; 2175. Third register access interface; 2176. Third memory read / write interface; 2177. Fourth register access interface; 2178. Fourth memory read / write interface; 2179. Fifth register access interface; 2180. Fifth memory read / write interface; 2181. Sixth register access interface; 2182. Sixth memory read / write interface; 2191. First memory access control interface; 2192. First arbitration interface; 2193. Second memory access control interface; 2194. Second arbitration interface; 2195. Third memory access control interface; 2196. Third arbitration interface. Detailed Implementation
[0037] The embodiments of this application will now be described with reference to the accompanying drawings. These embodiments are used to explain the technical solutions of this application, and not to limit the scope of protection of this application. Components and interfaces with the same names in different embodiments are referred to by the same reference numerals. Technical features in different embodiments can be combined with each other when there is no conflict.
[0038] In this application, "interface" refers to the signal connection boundary of a circuit or execution unit, used for transmitting instructions, status, data, or control signals. The interface itself does not perform decoding, computation, address generation, or arbitration functions. These functions are performed by the circuit connected to the interface. When an internal circuit's interface constitutes a corresponding interface of its unit, it means that both correspond to the same signal connection boundary, not that a separate interface is established between them.
[0039] In this application, a "thread block" refers to a group of threads executing the same kernel function. Thread blocks can share their corresponding execution context and resource allocation information. The current thread block is the one that issued a concurrency request instruction and has completed the matrix computation phase. The next thread block is another thread block determined from the list of thread blocks to be scheduled. The matrix computation phase includes matrix multiplication operations and matrix auxiliary operations related to the matrix multiplication results. Matrix auxiliary operations may include at least one of inverse quantization, post-summation, and accumulation. The post-processing phase may include at least one of root mean square normalization, positional encoding, and nonlinear operations.
[0040] Based on the aforementioned thread block definition and computation stage division, see [link to relevant documentation]. Figure 1 This application provides a kernel architecture 100 that supports concurrent execution of processing units and multiply-accumulate units.
[0041] The kernel architecture 100 includes a scheduler 110. The scheduler 110 includes a command FIFO buffer circuit 111, an instruction decoding circuit 112, a thread block selection circuit 113, and a concurrent dispatch circuit 114. The queue interaction interface 2101 of the command FIFO buffer circuit 111 is electrically connected to the queue access interface 2102 of the thread block selection circuit 113. The scheduling request interface 2103 of the instruction decoding circuit 112 is electrically connected to the request receiving interface 2104 of the thread block selection circuit 113. The decoding sending interface 2105 of the instruction decoding circuit 112 is electrically connected to the decoding receiving interface 2106 of the concurrent dispatch circuit 114. The dispatch control interface 2107 of the thread block selection circuit 113 is electrically connected to the scheduling control interface 2108 of the concurrent dispatch circuit 114.
[0042] The kernel architecture 100 also includes a multiply-accumulate unit 120. The matrix instruction receiving interface 2109 of the multiply-accumulate unit 120 is electrically connected to the first instruction dispatch interface 2110 of the concurrent dispatch circuit 114.
[0043] The kernel architecture 100 further includes at least two processing units. These at least two processing units include a first processing unit 130 and a second processing unit 140. The auxiliary instruction receiving interface 2111 of the first processing unit 130 is electrically connected to the second instruction distribution interface 2112 of the concurrent distribution circuit 114. The post-processing instruction receiving interface 2113 of the second processing unit 140 is electrically connected to the third instruction distribution interface 2114 of the concurrent distribution circuit 114.
[0044] The instruction decoding circuit 112 is used to parse the concurrent request instruction of the current thread block after the matrix calculation stage is completed. The thread block selection circuit 113 is used to check the queue status in response to the parsing result. When the queue is not empty, the thread block selection circuit 113 reads the command information and determines the next thread block according to the command information.
[0045] The concurrent distribution circuit 114 is used to distribute matrix multiplication instructions for the next thread block to the multiply-accumulate unit 120 via the first instruction distribution interface 2110. The concurrent distribution circuit 114 distributes matrix auxiliary operation instructions for the next thread block to the first processing unit 130 via the second instruction distribution interface 2112. The concurrent distribution circuit 114 distributes post-processing instructions for the current thread block to the second processing unit 140 via the third instruction distribution interface 2114, so that the matrix calculation stage of the next thread block and the post-processing stage of the current thread block overlap in execution time.
[0046] The multiply-add unit 120 performs matrix multiplication. The first processing unit 130 performs matrix auxiliary operations. The second processing unit 140 performs post-processing operations. The current thread block is the thread block to which the concurrent request instruction belongs. The next thread block is a thread block to be scheduled, distinct from the current thread block. The matrix calculation stage includes matrix multiplication and matrix auxiliary operations.
[0047] The queue status can include an empty state and a non-empty state, as well as a full state, a near-full state, or the number of commands that can be read. The thread block selection circuit 113 responds to concurrent request instructions, meaning that after receiving the parsing result output by the instruction decoding circuit 112, it begins to check whether there are any thread blocks available for scheduling. The thread block selection circuit 113 does not only select based on the thread block name, but can also combine the kernel identifier, thread block identifier, starting instruction address, and resource requirement information from the command information to determine the next thread block.
[0048] For example, thread block A becomes the current thread block after completing matrix multiplication, dequantization, and accumulation. Instruction decoding circuit 112 parses the concurrent request instruction of thread block A. When the command FIFO buffer circuit 111 stores the command information of thread block B, thread block selection circuit 113 determines thread block B as the next thread block. Concurrent dispatch circuit 114 dispatches the matrix multiplication instruction of thread block B to multiply-accumulate unit 120, dispatches the matrix auxiliary operation instruction of thread block B to first processing unit 130, and dispatches the post-processing instruction of thread block A to second processing unit 140. The matrix auxiliary operation of thread block B can be started in a timely manner according to its data dependency relationship with the matrix multiplication result, without requiring the three types of instructions to be issued in the same clock cycle.
[0049] The term "overlapping execution time" in this application refers to the fact that the post-processing stage of the current thread block and the matrix calculation stage of the next thread block share a common duration, without requiring that the two types of operations have the same start time, end time, or duration. The multiply-accumulate unit 120, the first processing unit 130, and the second processing unit 140 can each have independent pipelines, or they can proceed in a unified clock domain according to different valid signals.
[0050] As an extension architecture coupled with the aforementioned infrastructure, scheduler 110 can also establish a concurrency window record for the current thread block and the next thread block. The concurrency window record includes the identifiers of the two thread blocks, the allowed execution units, and the corresponding computation stages. The concurrency dispatch circuit 114 only opens the corresponding instruction dispatch interface when the decoded instruction matches the concurrency window record. This extension method can further constrain the allowed destination of instructions across thread blocks and reduce the possibility of incorrect dispatch when the thread block identifier is abnormal.
[0051] In this embodiment, the scheduler 110 determines the next thread block and allocates different operations to different execution units through three instruction dispatch paths, enabling the post-processing stage of the current thread block and the matrix calculation stage of the next thread block to execute concurrently. This structure helps reduce execution resource waiting and maintains the correspondence between instructions, thread blocks, and execution units.
[0052] Based on the aforementioned scheduler 110 receiving and storing the command information of the thread block to be scheduled, see [link to scheduler 110]. Figure 2 This application provides an internal structure for a command first-in-first-out (FIFO) cache circuit 111.
[0053] The command first-in-first-out (FIFO) cache circuit 111 includes a command storage array 1111, a read pointer register 1112, a write pointer register 1113, a queue state generation circuit 1114, and a read / write control circuit 1115.
[0054] The command receiving interface 2121 of the read / write control circuit 1115 is electrically connected to the command sending interface 2122 of the upper-level scheduling circuit, and is used to receive command information and write control signals from the thread block to be scheduled. The queue interaction interface 2101 of the read / write control circuit 1115 constitutes the queue interaction interface 2101 of the command first-in-first-out buffer circuit 111, and is used to transmit read control signals, queue status, and command information with the queue access interface 2102.
[0055] The storage access interface 2123 of the read / write control circuit 1115 is electrically connected to the storage read / write interface 2124 of the command storage array 1111. The read pointer access interface 2125 of the read / write control circuit 1115 is electrically connected to the read pointer read / write interface 2126 of the read pointer register 1112. The write pointer access interface 2127 of the read / write control circuit 1115 is electrically connected to the write pointer read / write interface 2128 of the write pointer register 1113.
[0056] The read pointer sending interface 2129 of the read pointer register 1112 is electrically connected to the first pointer receiving interface 2130 of the queue status generation circuit 1114. The write pointer sending interface 2131 of the write pointer register 1113 is electrically connected to the second pointer receiving interface 2132 of the queue status generation circuit 1114. The queue status sending interface 2133 of the queue status generation circuit 1114 is electrically connected to the queue status receiving interface 2134 of the read / write control circuit 1115.
[0057] The read / write control circuit 1115 is used to write command information into the command storage array 1111 according to the write pointer. After writing is completed, the read / write control circuit 1115 updates the write pointer. The read / write control circuit 1115 is also used to respond to the read control signal and read the earliest written but not yet read command information according to the read pointer. After the command information is received by the thread block selection circuit 113, the read / write control circuit 1115 updates the read pointer. The queue state generation circuit 1114 is used to generate a queue state based on the read pointer and the write pointer.
[0058] The command information may include thread block identifier, kernel function identifier, instruction start address, register resource requirements, and shared memory resource requirements. A write control signal indicates that the command information is valid. The thread block selection circuit 113 accepting the command information means that it completes the sampling of the command information and returns an acceptance confirmation or completes the corresponding read process to the read / write control circuit 1115.
[0059] Both the read pointer and the write pointer can include an array position field and a wraparound status field. The queue status generation circuit 1114 can distinguish between an empty and a full state based on the positional relationship between the two pointers and the wraparound status. Using this method, even if the read pointer and the write pointer point to the same array position, the wraparound status can be used to determine whether the command storage array 1111 is empty or full.
[0060] As an extension architecture coupled with the aforementioned infrastructure, the command information may also include a resource ready flag. The thread block selection circuit 113 only considers thread blocks with valid commands and ready resources as candidate thread blocks. When the head thread block in the queue does not yet meet the resource conditions, the scheduler 110 can retain its command information and wait for the resource to be released, or check other commands according to preset bypass reading rules. This extension method helps reduce the situation where a thread block is selected but then falls back due to insufficient resources.
[0061] This embodiment constructs a hardware command queue using a command storage array 1111, read / write pointers, and a queue state generation circuit 1114, enabling command information for thread blocks to be scheduled to enter the selection process in a predetermined order. Combined with the scheduling path described in the previous embodiment, this provides stable command information for the next thread block for concurrent execution across thread blocks.
[0062] Based on the fact that the aforementioned thread block selection circuit 113 has already been able to determine the instruction fetch object, see [link to relevant documentation]. Figure 3 This application provides an instruction acquisition and instruction routing structure.
[0063] The scheduler 110 also includes an instruction fetch circuit 115. The instruction fetch control interface 2141 of the thread block selection circuit 113 is electrically connected to the thread selection interface 2142 of the instruction fetch circuit 115. The instruction storage access interface 2143 of the instruction fetch circuit 115 is electrically connected to the instruction storage service interface 2144 of the instruction memory 116. The instruction provision interface 2145 of the instruction fetch circuit 115 is electrically connected to the instruction fetch interface 2146 of the instruction decoding circuit 112.
[0064] The thread block selection circuit 113 provides thread block selection information via the instruction fetch control interface 2141. The instruction fetch circuit 115 fetches the instruction for the corresponding thread block based on the thread block selection information. The instruction fetch circuit 115 provides the instruction to the instruction decoding circuit 112 via the instruction providing interface 2145. The thread block selection information is used to select the instruction fetch object between the current thread block and the next thread block.
[0065] The instruction decoding circuit 112 is used to provide decoding instructions carrying thread block identifiers and execution unit identifiers to the decoding sending interface 2105. The concurrent dispatch circuit 114 is used to receive decoding instructions through the decoding receiving interface 2106.
[0066] The concurrent dispatch circuit 114 is used to select an instruction path based on the execution unit identifier. When the execution unit identifier indicates the multiply-accumulate unit 120, the instruction received by the decoder-receiver interface 2106 is routed to the first instruction dispatch interface 2110. When the execution unit identifier indicates the first processing unit 130, the instruction received by the decoder-receiver interface 2106 is routed to the second instruction dispatch interface 2112. When the execution unit identifier indicates the second processing unit 140, the instruction received by the decoder-receiver interface 2106 is routed to the third instruction dispatch interface 2114. The concurrent dispatch circuit 114 retains the corresponding thread block identifier during instruction dispatch.
[0067] The thread block selection information may include a thread block identifier and instruction address information. The instruction fetch circuit 115 can read the program counter status corresponding to the thread block based on the thread block identifier and access the instruction memory 116 based on the instruction address information. The instruction memory 116 may be a kernel-local instruction memory or an instruction cache shared by multiple kernels.
[0068] The execution unit identifier indicates the target execution unit of the decoded instruction. The thread block identifier indicates the thread block to which the decoded instruction belongs. They serve different purposes. Even if the current thread block and the next thread block use the same opcode, the corresponding computation context can be distinguished based on the thread block identifier.
[0069] For example, the instruction fetching circuit 115 first fetches the post-processing instructions of the current thread block, and then fetches the matrix multiplication instructions and matrix auxiliary operation instructions of the next thread block. The instruction decoding circuit 112 retains the thread block identifier for each decoded instruction and adds the corresponding execution unit identifier. The concurrent dispatch circuit 114 selects the output path according to the execution unit identifier, and simultaneously passes the thread block identifier along with the instruction to the target execution unit.
[0070] As an extension architecture coupled with the aforementioned infrastructure, the decoding instructions can also carry instruction sequence identifiers. After receiving the decoding instructions, the target execution unit returns a reception status corresponding to the instruction sequence identifier. The concurrent dispatch circuit 114 retains the corresponding decoding instructions before obtaining the reception status. This extension method is beneficial for maintaining the correspondence between instructions and thread blocks when the execution unit is temporarily unable to receive instructions.
[0071] This embodiment establishes the connection between the instruction fetching object and the thread block through thread block selection information, and establishes the connection between the decoded instructions and the dispatch path through the thread block identifier and the execution unit identifier. This structure can provide hardware support for instruction fetching and accurate dispatch when the current thread block and the next thread block exist simultaneously.
[0072] Based on the premise that the current thread block and the next thread block can be processed by different execution units, see [link to relevant documentation]. Figure 4 This application provides a schematic diagram of a thread block selection circuit and an execution status transmission structure.
[0073] The thread block selection circuit 113 includes a selection control circuit 1131 and a scheduling status register 1132. The status access interface 2151 of the selection control circuit 1131 is electrically connected to the scheduling status read / write interface 2152 of the scheduling status register 1132.
[0074] The queue access interface 2102 of the selection control circuit 1131 constitutes the queue access interface 2102 of the thread block selection circuit 113. The request receiving interface 2104 of the selection control circuit 1131 constitutes the request receiving interface 2104 of the thread block selection circuit 113. The distribution control interface 2107 of the selection control circuit 1131 constitutes the distribution control interface 2107 of the thread block selection circuit 113.
[0075] The first execution status sending interface 2153 of the multiply-accumulate unit 120 is electrically connected to the first execution status receiving interface 2154 of the selection control circuit 1131. The second execution status sending interface 2155 of the first processing unit 130 is electrically connected to the second execution status receiving interface 2156 of the selection control circuit 1131. The third execution status sending interface 2157 of the second processing unit 140 is electrically connected to the third execution status receiving interface 2158 of the selection control circuit 1131.
[0076] The selection control circuit 1131 is used to update the scheduling status register 1132 according to the received execution status. The scheduling status register 1132 is used to record the matrix calculation completion status, post-processing completion status, and occupancy status of the second processing unit 140 of the thread block.
[0077] After the matrix calculation of the next thread block is completed, the selection control circuit 1131 checks the occupancy status. If the second processing unit 140 is occupied, the selection control circuit 1131 records the post-processing pending launch status of the next thread block. If the second processing unit 140 is idle, the selection control circuit 1131 sends a post-processing distribution permission via the distribution control interface 2107. The concurrent distribution circuit 114 receives the post-processing distribution permission via the scheduling control interface 2108 and distributes the post-processing instructions for the next thread block.
[0078] The selection control circuit 1131 is also used to check the queue status when responding to a concurrent request instruction. When the queue is empty, no new thread block is started, and the current thread block is allowed to continue executing the post-processing stage.
[0079] The matrix calculation completion status can be jointly determined by the matrix multiplication completion status of the multiply-add unit 120 and the matrix auxiliary operation completion status of the first processing unit 130. The post-processing completion status indicates whether the second processing unit 140 has completed the post-processing of the corresponding thread block. The occupancy status indicates whether the second processing unit 140 is currently bound to a certain thread block.
[0080] The post-processing pending launch state is not the post-processing completed state. The post-processing pending launch state indicates that the corresponding thread block has met the prerequisites for executing post-processing, but the second processing unit 140 is not yet available. After the second processing unit 140 becomes idle, the selection control circuit 1131 generates a post-processing distribution permission based on this state.
[0081] For example, if the post-processing of the current thread block A has not yet been completed, but the matrix calculation of the next thread block B has been completed, the selection control circuit 1131 does not immediately send a post-processing instruction to thread block B, but instead records the post-processing pending state of thread block B in the scheduling status register 1132. After the post-processing of thread block A is completed, the second processing unit 140 is released, and the selection control circuit 1131 then permits the post-processing instruction of thread block B to enter the third instruction dispatch interface 2114.
[0082] As an extension architecture coupled with the aforementioned infrastructure, the scheduling status register 1132 can also set the instruction issued state and execution completed state for each thread block. The selection control circuit 1131 generates a post-processing dispatch permission only when the post-processing pending state is valid and the instruction issued state is invalid, and updates the instruction issued state after receiving the instruction received state. This extension helps reduce the possibility of the same post-processing instruction being issued repeatedly.
[0083] This embodiment adapts the completion order of the two thread blocks by recording the execution progress and the occupancy status of the second processing unit 140. Combined with the instruction dispatch structure of the aforementioned embodiment, it can prevent post-processing resource conflicts while maintaining concurrent execution.
[0084] Based on the aforementioned three types of instructions being delivered to different execution units, see [link to relevant documentation]. Figure 5 The embodiments of this application provide the internal structure of the multiply-accumulate unit 120, the first processing unit 130, and the second processing unit 140.
[0085] The multiply-accumulate unit 120 includes a multiply-accumulate control circuit 121 and a multiply-accumulate array 122. The matrix instruction receiving interface 2109 of the multiply-accumulate control circuit 121 constitutes the matrix instruction receiving interface 2109 of the multiply-accumulate unit 120. The multiply-accumulate control transmitting interface 2161 of the multiply-accumulate control circuit 121 is electrically connected to the multiply-accumulate control receiving interface 2162 of the multiply-accumulate array 122. The multiply-accumulate control circuit 121 generates multiply-accumulate control signals according to matrix multiplication instructions. The multiply-accumulate array 122 performs matrix multiplication operations in response to the multiply-accumulate control signals.
[0086] The first processing unit 130 includes a first arithmetic circuit 131. The auxiliary instruction receiving interface 2111 of the first arithmetic circuit 131 constitutes the auxiliary instruction receiving interface 2111 of the first processing unit 130. The first arithmetic circuit 131 is dedicated to performing at least one of the following operations related to matrix multiplication: inverse quantization, post-summation, and accumulation.
[0087] The second processing unit 140 includes a second arithmetic circuit 141. The post-processing instruction receiving interface 2113 of the second arithmetic circuit 141 constitutes the post-processing instruction receiving interface 2113 of the second processing unit 140. The second arithmetic circuit 141 is used to execute post-processing instructions or general calculation instructions. The post-processing instructions or general calculation instructions include at least one of root mean square normalization instructions, position encoding instructions, and nonlinear arithmetic instructions.
[0088] The second arithmetic circuit 141 and the first arithmetic circuit 131 are different hardware arithmetic circuits. When the second arithmetic circuit 141 executes the post-processing instructions of the current thread block, it does not occupy the first arithmetic circuit 131 and the multiply-accumulate array 122.
[0089] The matrix instruction receiving interface 2109 of the multiply-accumulate control circuit 121 and the matrix instruction receiving interface 2109 of the multiply-accumulate unit 120 share the same signal connection boundary. The auxiliary instruction receiving interface 2111 and the post-processing instruction receiving interface 2113 adopt the same interface hierarchy. Therefore, "configuration" does not imply the existence of two serially connected interfaces with the same name between the internal circuit and its corresponding execution unit.
[0090] Matrix multiplication instructions can include matrix size information, data type information, and data position indication information. The multiply-accumulate control circuit 121 generates array enable, data strobe, and accumulation control signals based on the matrix multiplication instructions. The multiply-accumulate array 122 can be composed of multiple multipliers and adders, and processes the input data in a matrix block manner.
[0091] Dequantization is used to convert quantized data into the data representation required for subsequent calculations. Post-summation is used to sum the local results generated by matrix multiplication. Accumulation is used to combine multiple local calculation results. These matrix-aided operations, together with matrix multiplication, form the matrix computation process.
[0092] Root mean square normalization is used to adjust the data scale based on the root mean square characteristics of the input data. Position encoding is used to incorporate position information into the data representation. Nonlinear operations may include activation operations or other nonlinear transformations. The second arithmetic circuit 141 can also execute general-purpose computation instructions independent of the multiply-accumulate array 122.
[0093] As an extension architecture coupled with the aforementioned infrastructure, the three types of instruction receiving interfaces can each be configured with local instruction buffers and ready handshake logic. The concurrent dispatch circuit 114 completes instruction handover only when the corresponding local instruction buffer has available space. The execution unit starts the corresponding operation based on the data ready state. This extension method can decouple the instruction dispatch timing from the data preparation timing and reduce the impact of temporary blockage of the execution unit on other instruction paths.
[0094] In this embodiment, different hardware operation circuits are used to perform matrix multiplication, matrix auxiliary operation and post-processing operation respectively. When the next thread block performs matrix calculation, the current thread block can use the second operation circuit 141 to perform post-processing, thereby reducing the contention of different operation links for the same processing resources.
[0095] Based on the aforementioned execution units processing data from different thread blocks, see [link to relevant documentation]. Figure 6 This application provides a physically isolated register structure.
[0096] The kernel architecture 100 also includes a first register file 150, a second register file 160, and a third register file 170 that are physically independent of each other.
[0097] The first register access interface 2171 of the multiply-accumulate unit 120 is electrically connected to the first storage read / write interface 2172 of the first register file 150. The second register access interface 2173 of the first processing unit 130 is electrically connected to the second storage read / write interface 2174 of the second register file 160. The third register access interface 2175 of the second processing unit 140 is electrically connected to the third storage read / write interface 2176 of the third register file 170.
[0098] During the concurrent execution of the current thread block and the next thread block, the multiply-accumulate unit 120 accesses the matrix multiplication operation data of the next thread block via the first register access interface 2171. The first processing unit 130 accesses the matrix auxiliary operation data of the next thread block via the second register access interface 2173. The second processing unit 140 accesses the post-processing operation data of the current thread block via the third register access interface 2175.
[0099] Physical independence means that the first register file 150, the second register file 160, and the third register file 170 each have independent storage units and corresponding access paths. Each register file can be located within the same chip, but different register files do not reuse the same storage unit.
[0100] For example, the first register file 150 stores the matrix input data, weight data, and intermediate matrix multiplication results of the next thread block. The second register file 160 stores the dequantization parameters and accumulation results of the next thread block. The third register file 170 stores the post-processing input, intermediate results, and output of the current thread block. Thus, even if three execution units perform corresponding operations simultaneously, data will not be overwritten due to accessing the same register storage unit.
[0101] As an extension architecture coupled to the aforementioned infrastructure, scheduler 110 can maintain register file allocation records. These records represent the binding relationships between thread blocks and register files. Scheduler 110 only releases the corresponding binding relationships after both the matrix computation and post-processing stages of a thread block are completed. This extension prevents the register files containing intermediate results still used by the post-processing stages from being pre-allocated to other thread blocks.
[0102] This embodiment utilizes physically independent register files to provide isolated data spaces for the three execution units. By combining different hardware execution resources, it can simultaneously reduce computational resource contention and register access conflicts, providing stable data storage conditions for concurrent execution across thread blocks.
[0103] Based on the aforementioned requirement that different execution units need to isolate and store computational data, see [link to relevant documentation]. Figure 7 This application provides a logically isolated register structure.
[0104] The kernel architecture 100 also includes a register storage array 180. The fourth register access interface 2177 of the multiply-accumulate unit 120 is electrically connected to the fourth storage read / write interface 2178 of the register storage array 180. The fifth register access interface 2179 of the first processing unit 130 is electrically connected to the fifth storage read / write interface 2180 of the register storage array 180. The sixth register access interface 2181 of the second processing unit 140 is electrically connected to the sixth storage read / write interface 2182 of the register storage array 180.
[0105] The register storage array 180 has non-overlapping first address range 1801, second address range 1802, and third address range 1803. A fourth storage read / write interface 2178 is used to access the first address range. A fifth storage read / write interface 2180 is used to access the second address range. A sixth storage read / write interface 2182 is used to access the third address range.
[0106] During the concurrent execution of the current thread block and the next thread block, the multiply-accumulate unit 120 accesses the matrix multiplication operation data of the next thread block via the fourth register access interface 2177. The first processing unit 130 accesses the matrix auxiliary operation data of the next thread block via the fifth register access interface 2179. The second processing unit 140 accesses the post-processing operation data of the current thread block via the sixth register access interface 2181.
[0107] In this context, non-overlapping address ranges refer to any address belonging to only one of the first, second, and third address ranges during the same allocation period. The capacities of the three address ranges can be the same, or they can be set separately based on the data volume of matrix multiplication, matrix auxiliary operations, and post-processing operations.
[0108] The fourth storage read / write interface 2178, the fifth storage read / write interface 2180, and the sixth storage read / write interface 2182 can be independent access ports of the register storage array 180, or they can be logical access channels formed after port arbitration. Regardless of the physical implementation used, each access interface is constrained to its corresponding address range.
[0109] For example, a portion of the address space of register storage array 180 is allocated to matrix multiplication operations in the next thread block, another portion to matrix auxiliary operations in the next thread block, and the remainder to post-processing operations in the current thread block. Each execution unit carries the corresponding thread block identifier and address range information when generating an access request, thereby limiting data access to the appropriate region.
[0110] As an extension architecture coupled with the aforementioned infrastructure, each storage read / write interface can also be connected to an address boundary checking circuit. The address boundary checking circuit compares the access address with the corresponding address range, allowing read / write operations only if the access address falls within the corresponding address range. This extension further reduces the possibility of cross-region access caused by address generation anomalies.
[0111] This embodiment achieves logical isolation by dividing the shared register storage array 180 into non-overlapping address ranges. This structure can maintain the data boundary between the current thread block and the next thread block while sharing physical storage resources, making it convenient to select the register implementation method according to kernel area and storage capacity requirements.
[0112] Based on the aforementioned ability of multiple execution units to concurrently perform computations, see [link to relevant documentation]. Figure 8 This application provides a memory access launch control structure.
[0113] The kernel architecture 100 also includes a memory access issue control circuit 190. The first memory access control interface 2191 of the multiply-accumulate unit 120 is electrically connected to the first arbitration interface 2192 of the memory access issue control circuit 190. The second memory access control interface 2193 of the first processing unit 130 is electrically connected to the second arbitration interface 2194 of the memory access issue control circuit 190. The third memory access control interface 2195 of the second processing unit 140 is electrically connected to the third arbitration interface 2196 of the memory access issue control circuit 190.
[0114] The first memory access control interface 2191 and the first arbitration interface 2192 are used to transmit memory access requests and corresponding authorization signals from the multiply-accumulate unit 120. The second memory access control interface 2193 and the second arbitration interface 2194 are used to transmit memory access requests and corresponding authorization signals from the first processing unit 130. The third memory access control interface 2195 and the third arbitration interface 2196 are used to transmit memory access requests and corresponding authorization signals from the second processing unit 140.
[0115] The memory-to-issue control circuit 190 is used for column-level issue arbitration. During a column-level issue cycle authorized by the multiply-accumulate unit 120, the memory-to-issue control circuit 190 prohibits the first processing unit 130 from issuing load or store instructions. The memory-to-issue control circuit 190 also prohibits the second processing unit 140 from issuing load or store instructions during that cycle. During a column-level issue cycle authorized by either processing unit, the memory-to-issue control circuit 190 prohibits the multiply-accumulate unit 120 from issuing load or store instructions.
[0116] A single multiply-accumulate operation of the multiply-accumulate unit 120 lasts for at least two column-level transmit cycles. During a column-level transmit cycle in which the multiply-accumulate unit 120 does not issue a load instruction or a store instruction, the memory access transmit control circuit 190 provides an authorization signal to the first processing unit 130 or the second processing unit 140 that has a memory access request.
[0117] The column-level issue cycle is the control granularity at which the memory issue control circuit 190 authorizes load or store instructions. A single multiply-accumulate operation can process different columns or different data blocks of the matrix sequentially within multiple column-level issue cycles, but not every column-level issue cycle requires memory access.
[0118] For example, after the multiply-accumulate unit 120 reads the next set of matrix data in one column-level transmit cycle, it can continue to perform multiply-accumulate operations in multiple subsequent column-level transmit cycles. If there is no loading or storage request from the multiply-accumulate unit 120 in one of the cycles, the memory access transmit control circuit 190 can authorize the first processing unit 130 to read the matrix auxiliary operation data or authorize the second processing unit 140 to read and write the post-processed data in that cycle.
[0119] When the first processing unit 130 and the second processing unit 140 simultaneously submit memory access requests, the memory access launch control circuit 190 can select one of the requests according to a fixed priority, a round-robin order, or a request waiting time. This selection rule does not change the mutually exclusive relationship between memory access requests from the processing unit side and memory access requests from the multiply-accumulate unit side within the same column-level launch cycle.
[0120] As an extension architecture coupled with the aforementioned infrastructure, the memory access issue control circuit 190 can record the continuous waiting state of each memory access request. When a memory access request on the processing unit side continues to wait until a set condition is met, and the multiply-accumulate unit 120 does not have a memory access instruction that must be issued in the current column-level issue cycle, the corresponding processing unit is given priority authorization. This extension method is beneficial for balancing multiply-accumulate data supply and post-processing data access, reducing the situation where one side's request continues to wait.
[0121] This embodiment prevents the multiply-accumulate unit 120 and the processing unit from competing for the memory access launch path in the same control cycle by using column-level launch arbitration, and uses the non-memory access cycle in the multiply-accumulate operation to process other memory access requests, thereby enabling computational concurrency and orderly memory access control to cooperate with each other.
[0122] Based on the aforementioned kernel architecture, which already includes command caching, thread block selection, and multi-execution unit dispatch paths, see [link to kernel architecture]. Figure 9 This application provides a kernel scheduling method.
[0123] In step S901, the matrix multiplication operation of the current thread block is performed by the multiply-add unit, and the matrix auxiliary operation of the current thread block is performed by the first processing unit.
[0124] In step S902, after the matrix calculation of the current thread block is completed, the instruction decoding circuit parses the concurrent request instruction of the current thread block and transmits the parsing result to the request receiving interface through the scheduling request interface.
[0125] In step S903, the thread block selection circuit responds to the parsing result and accesses the queue interaction interface via the queue access interface to obtain the queue status. When the queue is not empty, the thread block selection circuit reads the command information of the thread block to be scheduled and determines the next thread block based on the command information.
[0126] In step S904, the thread block selection circuit provides thread block scheduling information to the scheduling control interface via the distribution control interface, and the instruction decoding circuit provides decoding instructions to the decoding receiving interface via the decoding sending interface.
[0127] In step S905, the concurrent dispatch circuit, based on the thread block scheduling information, dispatches the matrix multiplication instruction of the next thread block to the matrix instruction receiving interface via the first instruction dispatch interface, the matrix auxiliary operation instruction of the next thread block to the auxiliary instruction receiving interface via the second instruction dispatch interface, and the post-processing instruction of the current thread block to the post-processing instruction receiving interface via the third instruction dispatch interface.
[0128] In step S906, the multiply-accumulate unit performs the matrix multiplication operation of the next thread block, the first processing unit performs the matrix auxiliary operation of the next thread block, and the second processing unit performs the post-processing operation of the current thread block. The scheduler arranges the instruction distribution timing so that the matrix calculation stage of the next thread block and the post-processing stage of the current thread block overlap in execution time.
[0129] The completion of the matrix calculation phase in the current thread block can be determined by the thread block selection circuit after the multiply-accumulate unit and the first processing unit output their respective completion statuses. Concurrent request instructions can be located at the instruction boundary between the matrix calculation phase and the post-processing phase, or they can be prefetched before the matrix calculation phase ends and take effect after the matrix calculation completion condition is met.
[0130] Thread block scheduling information may include the next thread block identifier, the current thread block identifier, the allowed instruction types, and execution unit occupancy information. Decoding instructions may include the opcode, source operand information, destination operand information, thread block identifier, and execution unit identifier. The concurrent dispatch circuit combines the thread block scheduling information and the decoding instructions to determine the corresponding output path.
[0131] For example, after thread block A completes its matrix calculation, the thread block selection circuit selects thread block B from the command first-in-first-out (FIFO) cache circuit. The scheduler first sends the post-processing instructions of thread block A to the second processing unit, and then sends the matrix multiplication instructions of thread block B to the multiply-accumulate unit. The matrix auxiliary operation instructions of thread block B enter the first processing unit after the required data is ready. The execution time overlap occurs whenever the post-processing duration of thread block A overlaps with the matrix calculation duration of thread block B.
[0132] As an extended scheduling method coupled with the aforementioned infrastructure, the scheduler can pre-fetch instructions for the next thread block and open the corresponding instruction dispatch interface after the matrix calculation completion status of the current thread block is valid. The pre-fetched instructions still retain the next thread block identifier and do not enter the execution unit before being opened. This method can shorten the control waiting time between the completion of matrix calculation and the concurrent start of cross-thread blocks.
[0133] This embodiment triggers the selection of the next thread block and the dispatch of three types of instructions through hardware states, enabling cross-thread block pipelines to operate independently of developers orchestrating low-level instructions line by line. Combined with execution resource allocation and thread block identifier preservation mechanisms, it can reduce serial waiting while maintaining instruction correspondence.
[0134] Based on the premise that the current thread block and the next thread block have entered a concurrent execution state, see [link to relevant documentation]. Figure 10 This application provides a method for post-processing scheduling based on the execution completion order.
[0135] In step S1001, the selection control circuit updates the matrix calculation completion status in the scheduling status register based on the execution status received by the first execution status receiving interface and the second execution status receiving interface. The selection control circuit also updates the post-processing completion status and the occupancy status of the second processing unit based on the execution status received by the third execution status receiving interface.
[0136] In step S1002, when the post-processing stage of the current thread block is completed first, the selection control circuit updates the occupancy status of the second processing unit to idle. The multiply-accumulate unit continues to execute the matrix multiplication operation that has not yet been completed in the next thread block, and the first processing unit continues to execute the matrix auxiliary operation that has not yet been completed in the next thread block.
[0137] In step S1003, when the matrix calculation stage of the next thread block is completed first, the selection control circuit checks the occupancy status of the second processing unit. If the second processing unit is still occupied by the current thread block, the selection control circuit records the post-processing pending launch status of the next thread block.
[0138] In step S1004, after the matrix calculation stage of the next thread block is completed, the selection control circuit sends a post-processing distribution permission to the scheduling control interface via the distribution control interface when the second processing unit is idle. The concurrent distribution circuit responds to the post-processing distribution permission and distributes the post-processing instructions for the next thread block to the post-processing instruction receiving interface via the third instruction distribution interface. The second processing unit then executes the post-processing operation of the next thread block.
[0139] In this context, "the post-processing of the current thread block completes first" means that the post-processing completion status of the current thread block takes effect before the matrix calculation completion status of the next thread block. At this point, the second processing unit can become idle, while the multiply-accumulate unit and the first processing unit continue processing the unfinished matrix calculations of the next thread block, without waiting for the second processing unit to perform other operations.
[0140] The matrix calculation phase of the next thread block completes first, meaning that when the matrix calculation completion status of the next thread block becomes effective, the post-processing completion status of the current thread block has not yet become effective. At this time, the next thread block already has the data conditions to enter the post-processing phase, but the second processing unit is still occupied by the current thread block, so it is necessary to record the post-processing pending launch status.
[0141] For example, when the matrix calculation of thread block B is completed first, the selection control circuit marks thread block B as a post-processing instruction to be issued. After thread block A completes its post-processing, the third execution status receiving interface receives the completion status. The selection control circuit updates the second processing unit to idle and checks the post-processing instruction to be issued status. When thread block B is in the issuing state, the selection control circuit generates a post-processing distribution permission, allowing the post-processing instructions of thread block B to enter the second processing unit.
[0142] As an extension coupled with the aforementioned scheduling method, the selection control circuit can configure the occupancy state update, post-processing pre-launch state reading, and post-processing distribution permit generation as an ordered state transition process. The next state transition is only allowed after the previous state is written to the scheduling state register. This extension reduces the possibility of duplicate or missed launches when the execution completion state and resource release state change simultaneously.
[0143] This embodiment controls the release and reallocation of the second processing unit based on the actual completion order of the current thread block and the next thread block. When the current thread block completes its processing first, the matrix calculation of the next thread block can continue; when the next thread block completes its matrix calculation first, its subsequent processing instructions are reserved until the second processing unit is idle before being issued. Therefore, two thread blocks can be continuously connected while maintaining a clear resource occupancy relationship.
[0144] In one combined implementation, the scheduler employs a command readiness flag, a concurrent window record, an instruction sequence identifier, and a scheduling status register. The command readiness flag is used to filter the next thread block that meets the resource requirements. The concurrent window record is used to limit the execution units that can be used by the current thread block and the next thread block. The instruction sequence identifier is used to maintain the correspondence between instruction sending and receiving. The scheduling status register is used to record the completion status of operations and the occupancy status of the second processing unit.
[0145] In this combined implementation, the thread block selection circuit first reads the resource-ready command information from the command FIFO cache circuit, and then establishes a concurrent window for the current thread block and the next thread block. The instruction decoding circuit adds a thread block identifier, execution unit identifier, and instruction sequence identifier to the decoded instructions. The concurrent dispatch circuit opens the corresponding instruction dispatch interface according to the concurrent window record. After receiving the instruction, the execution unit reports back the reception status, and the selection control circuit updates the scheduling status register accordingly.
[0146] This combined implementation method establishes an interconnected hardware control process encompassing command selection, instruction fetching, instruction distribution, execution confirmation, and state switching. Even when instructions from different thread blocks are interleaved within the scheduler, the correspondence can be maintained based on the thread block identifier and instruction sequence identifier, and the launch timing can be determined based on the execution unit's occupancy status.
[0147] In another combined implementation, at least a portion of the first, second, and third register files can be replaced by independent address ranges of the register storage array. For example, multiply-accumulate units with high data bandwidth requirements use an independent first register file, while the first and second processing units use non-overlapping second and third address ranges in the register storage array, respectively.
[0148] This hybrid storage approach still maintains the data boundaries between different execution units. The scheduler records the correspondence between thread blocks and physical register files or logical address ranges, and releases this correspondence after the matrix calculation and post-processing stages of a thread block are completed. Therefore, register resources can be configured according to the data bandwidth and storage capacity requirements of different computational stages.
[0149] In another combined implementation, the memory-issue control circuit works in conjunction with the scheduling status register. The memory-issue control circuit provides the memory wait status to the selection control circuit. When determining the next thread block, the selection control circuit can use the execution unit occupancy status, register availability status, and memory wait status as selection criteria.
[0150] For example, when a candidate thread block needs to load a large amount of matrix data immediately, while the current thread block's post-processing still has continuous memory access requests, the thread block selection circuit can postpone starting the candidate thread block or select another thread block with lower memory access requirements. This approach ensures that cross-thread block concurrent scheduling considers not only computational resources but also the availability of memory access paths, which helps reduce waiting caused by memory congestion after concurrent startup. Those skilled in the art should understand that the above circuit can be implemented using combinational logic, sequential logic, registers, memory arrays, or combinations thereof. Multiple functional circuits can be integrated into the same physical module or set up separately. The circuit name is used to describe its function and signal relationships and does not necessarily limit its packaging form or physical boundaries.
[0151] The above embodiments are only used to illustrate the technical concept of this application. Without departing from the technical concept of this application, adaptive adjustments to the interface bit width, queue depth, number of execution units, register capacity, instruction encoding method, and status recording method can all be used as specific embodiments of this application.
Claims
1. A kernel architecture that supports concurrent execution of processing units and multiply-accumulate units, characterized in that, include: The scheduler (110) includes a command first-in-first-out (FIFO) buffer circuit (111), an instruction decoding circuit (112), a thread block selection circuit (113), and a concurrent dispatch circuit (114). The queue interaction interface (2101) of the command FIFO buffer circuit (111) is electrically connected to the queue access interface (2102) of the thread block selection circuit (113). The scheduling request interface (2103) of the instruction decoding circuit (112) is electrically connected to the request receiving interface (2104) of the thread block selection circuit (113). The decoding sending interface (2105) of the instruction decoding circuit (112) is electrically connected to the decoding receiving interface (2106) of the concurrent dispatch circuit (114). The dispatch control interface (2107) of the thread block selection circuit (113) is electrically connected to the scheduling control interface (2108) of the concurrent dispatch circuit (114). The multiply-accumulate unit (120) has a matrix instruction receiving interface (2109) that is electrically connected to the first instruction distribution interface (2110) of the concurrent distribution circuit (114); At least two processing units, including a first processing unit (130) and a second processing unit (140); the auxiliary instruction receiving interface (2111) of the first processing unit (130) is electrically connected to the second instruction distribution interface (2112) of the concurrent distribution circuit (114); the post-processing instruction receiving interface (2113) of the second processing unit (140) is electrically connected to the third instruction distribution interface (2114) of the concurrent distribution circuit (114); The instruction decoding circuit (112) is used to parse the concurrent request instruction after the matrix calculation stage of the current thread block is completed; the thread block selection circuit (113) is used to check the queue status in response to the parsing result; when the queue is not empty, the thread block selection circuit (113) reads the command information; the thread block selection circuit (113) determines the next thread block according to the command information; the concurrent distribution circuit (114) is used to distribute the matrix multiplication instruction of the next thread block to the multiply-accumulate unit (120) through the first instruction distribution interface (2110); and to the first processing unit (120) through the second instruction distribution interface (2112). 130) Distribute the matrix auxiliary operation instruction of the next thread block; distribute the post-processing instruction of the current thread block to the second processing unit (140) via the third instruction distribution interface (2114) so that the matrix calculation stage of the next thread block and the post-processing stage of the current thread block overlap in execution time; the multiply-accumulate unit (120) performs matrix multiplication operation; the first processing unit (130) performs matrix auxiliary operation; the second processing unit (140) performs post-processing operation; the thread block is a group of threads that execute the same kernel function; the current thread block is the thread block to which the concurrent request instruction belongs; The next thread block is a thread block to be scheduled that is different from the current thread block; the matrix calculation step includes matrix multiplication and matrix auxiliary operations.
2. The kernel architecture according to claim 1, characterized in that, The command first-in-first-out cache circuit (111) includes a command storage array (1111), a read pointer register (1112), a write pointer register (1113), a queue state generation circuit (1114), and a read-write control circuit (1115). The command receiving interface (2121) of the read / write control circuit (1115) is electrically connected to the command sending interface (2122) of the upper-level scheduling circuit, and is used to receive command information and write control signals of the thread block to be scheduled; the queue interaction interface (2101) of the read / write control circuit (1115) constitutes the queue interaction interface (2101) of the command first-in-first-out buffer circuit (111), and is used to transmit read control signals, queue status and command information with the queue access interface (2102); The storage access interface (2123) of the read / write control circuit (1115) is electrically connected to the storage read / write interface (2124) of the command storage array (1111); the read pointer access interface (2125) of the read / write control circuit (1115) is electrically connected to the read pointer read / write interface (2126) of the read pointer register (1112); the write pointer access interface (2127) of the read / write control circuit (1115) is electrically connected to the write pointer read / write interface (2128) of the write pointer register (1113); The read pointer sending interface (2129) of the read pointer register (1112) is electrically connected to the first pointer receiving interface (2130) of the queue state generation circuit (1114); the write pointer sending interface (2131) of the write pointer register (1113) is electrically connected to the second pointer receiving interface (2132) of the queue state generation circuit (1114); the queue state sending interface (2133) of the queue state generation circuit (1114) is electrically connected to the queue state receiving interface (2134) of the read / write control circuit (1115); The read / write control circuit (1115) is used to write the command information into the command storage array (1111) according to the write pointer; after the writing is completed, the read / write control circuit (1115) updates the write pointer; the read / write control circuit (1115) is also used to respond to the read control signal and read the earliest written and yet unread command information according to the read pointer; after the command information is accepted by the thread block selection circuit (113), the read / write control circuit (1115) updates the read pointer; the queue state generation circuit (1114) is used to generate the queue state according to the read pointer and the write pointer.
3. The kernel architecture according to claim 1 or 2, characterized in that, The scheduler (110) also includes an instruction acquisition circuit (115); The instruction fetch control interface (2141) of the thread block selection circuit (113) is electrically connected to the thread selection interface (2142) of the instruction fetch circuit (115); the instruction storage access interface (2143) of the instruction fetch circuit (115) is electrically connected to the instruction storage service interface (2144) of the instruction memory (116); the instruction provision interface (2145) of the instruction fetch circuit (115) is electrically connected to the instruction fetch interface (2146) of the instruction decoding circuit (112). The thread block selection circuit (113) is used to provide thread block selection information via the instruction fetch control interface (2141); the instruction fetching circuit (115) is used to fetch the instruction of the corresponding thread block according to the thread block selection information; the instruction fetching circuit (115) provides the instruction to the instruction decoding circuit (112) through the instruction providing interface (2145); the thread block selection information is used to select the instruction fetching object between the current thread block and the next thread block; The instruction decoding circuit (112) is used to provide the decoding sending interface (2105) with a decoding instruction carrying a thread block identifier and an execution unit identifier; the concurrent dispatch circuit (114) is used to receive the decoding instruction through the decoding receiving interface (2106); The concurrent distribution circuit (114) is used to select an instruction path according to the execution unit identifier; when the execution unit identifier indicates the multiply-accumulate unit (120), the instruction received by the decoder receiving interface (2106) is routed to the first instruction distribution interface (2110); when the execution unit identifier indicates the first processing unit (130), the instruction received by the decoder receiving interface (2106) is routed to the second instruction distribution interface (2112); when the execution unit identifier indicates the second processing unit (140), the instruction received by the decoder receiving interface (2106) is routed to the third instruction distribution interface (2114); the concurrent distribution circuit (114) retains the corresponding thread block identifier when distributing instructions.
4. The kernel architecture according to claim 1 or 2, characterized in that, The thread block selection circuit (113) includes a selection control circuit (1131) and a scheduling status register (1132). The status access interface (2151) of the selection control circuit (1131) is electrically connected to the scheduling status read / write interface (2152) of the scheduling status register (1132); The queue access interface (2102) of the selection control circuit (1131) constitutes the queue access interface (2102) of the thread block selection circuit (113); the request receiving interface (2104) of the selection control circuit (1131) constitutes the request receiving interface (2104) of the thread block selection circuit (113); the distribution control interface (2107) of the selection control circuit (1131) constitutes the distribution control interface (2107) of the thread block selection circuit (113). The first execution status sending interface (2153) of the multiply-accumulate unit (120) is electrically connected to the first execution status receiving interface (2154) of the selection control circuit (1131); the second execution status sending interface (2155) of the first processing unit (130) is electrically connected to the second execution status receiving interface (2156) of the selection control circuit (1131); the third execution status sending interface (2157) of the second processing unit (140) is electrically connected to the third execution status receiving interface (2158) of the selection control circuit (1131). The selection control circuit (1131) is used to update the scheduling status register (1132) according to the received execution status; the scheduling status register (1132) is used to record the matrix calculation completion status, post-processing completion status and occupancy status of the thread block and the second processing unit (140); After the matrix calculation of the next thread block is completed, the selection control circuit (1131) checks the occupancy status; if the second processing unit (140) is occupied, the selection control circuit (1131) records the post-processing pending launch status of the next thread block; if the second processing unit (140) is idle, the selection control circuit (1131) sends a post-processing distribution permission through the distribution control interface (2107); the concurrent distribution circuit (114) receives the post-processing distribution permission through the scheduling control interface (2108) and distributes the post-processing instructions of the next thread block; The selection control circuit (1131) is also used to check the queue status when responding to the concurrent request instruction; when the queue is empty, a new thread block is not started, and the current thread block is allowed to continue to execute the post-processing stage.
5. The kernel architecture according to claim 1 or 2, characterized in that, The multiply-accumulate unit (120) includes a multiply-accumulate control circuit (121) and a multiply-accumulate array (122); The matrix instruction receiving interface (2109) of the multiply-accumulate control circuit (121) constitutes the matrix instruction receiving interface (2109) of the multiply-accumulate unit (120); the multiply-accumulate control sending interface (2161) of the multiply-accumulate control circuit (121) is electrically connected to the multiply-accumulate control receiving interface (2162) of the multiply-accumulate array (122); the multiply-accumulate control circuit (121) is used to generate multiply-accumulate control signals according to matrix multiplication instructions; the multiply-accumulate array (122) is used to perform matrix multiplication operations in response to the multiply-accumulate control signals; The first processing unit (130) includes a first arithmetic circuit (131); the auxiliary instruction receiving interface (2111) of the first arithmetic circuit (131) constitutes the auxiliary instruction receiving interface (2111); the first arithmetic circuit (131) is dedicated to performing at least one of matrix multiplication-related inverse quantization, post-summation, and accumulation operations; The second processing unit (140) includes a second arithmetic circuit (141); the post-processing instruction receiving interface (2113) of the second arithmetic circuit (141) constitutes the post-processing instruction receiving interface (2113); the second arithmetic circuit (141) is used to execute post-processing instructions or general calculation instructions, the post-processing instructions or general calculation instructions include at least one of root mean square normalization instructions, position encoding instructions and nonlinear operation instructions; the second arithmetic circuit (141) and the first arithmetic circuit (131) are different hardware arithmetic circuits; when the second arithmetic circuit (141) executes the post-processing instructions of the current thread block, it does not occupy the first arithmetic circuit (131) and the multiply-accumulate array (122).
6. The kernel architecture according to claim 1 or 2, characterized in that, The kernel architecture (100) also includes a first register file (150), a second register file (160), and a third register file (170) that are physically independent of each other. The first register access interface (2171) of the multiply-accumulate unit (120) is electrically connected to the first storage read / write interface (2172) of the first register file (150); the second register access interface (2173) of the first processing unit (130) is electrically connected to the second storage read / write interface (2174) of the second register file (160); the third register access interface (2175) of the second processing unit (140) is electrically connected to the third storage read / write interface (2176) of the third register file (170). During the concurrent execution of the current thread block and the next thread block, the multiply-accumulate unit (120) accesses the matrix multiplication operation data of the next thread block through the first register access interface (2171); the first processing unit (130) accesses the matrix auxiliary operation data of the next thread block through the second register access interface (2173); and the second processing unit (140) accesses the post-processing operation data of the current thread block through the third register access interface (2175).
7. The kernel architecture according to claim 1 or 2, characterized in that, The kernel architecture (100) also includes a register storage array (180). The fourth register access interface (2177) of the multiply-accumulate unit (120) is electrically connected to the fourth storage read / write interface (2178) of the register storage array (180); the fifth register access interface (2179) of the first processing unit (130) is electrically connected to the fifth storage read / write interface (2180) of the register storage array (180); the sixth register access interface (2181) of the second processing unit (140) is electrically connected to the sixth storage read / write interface (2182) of the register storage array (180). The register storage array (180) has a first address range, a second address range, and a third address range that do not overlap; the fourth storage read / write interface (2178) is used to access the first address range; the fifth storage read / write interface (2180) is used to access the second address range; and the sixth storage read / write interface (2182) is used to access the third address range. During the concurrent execution of the current thread block and the next thread block, the multiply-accumulate unit (120) accesses the matrix multiplication operation data of the next thread block via the fourth register access interface (2177); the first processing unit (130) accesses the matrix auxiliary operation data of the next thread block via the fifth register access interface (2179); and the second processing unit (140) accesses the post-processing operation data of the current thread block via the sixth register access interface (2181).
8. The kernel architecture according to claim 1 or 2, characterized in that, The kernel architecture (100) also includes a memory access issue control circuit (190). The first memory access control interface (2191) of the multiply-accumulate unit (120) is electrically connected to the first arbitration interface (2192) of the memory access issue control circuit (190); the second memory access control interface (2193) of the first processing unit (130) is electrically connected to the second arbitration interface (2194) of the memory access issue control circuit (190); the third memory access control interface (2195) of the second processing unit (140) is electrically connected to the third arbitration interface (2196) of the memory access issue control circuit (190). The first memory access control interface (2191) and the first arbitration interface (2192) are used to transmit the memory access request and corresponding authorization signal of the multiply-accumulate unit (120); the second memory access control interface (2193) and the second arbitration interface (2194) are used to transmit the memory access request and corresponding authorization signal of the first processing unit (130); the third memory access control interface (2195) and the third arbitration interface (2196) are used to transmit the memory access request and corresponding authorization signal of the second processing unit (140). The memory access launch control circuit (190) is used to perform column-level launch arbitration; during the column-level launch cycle authorized by the multiply-accumulate unit (120), the memory access launch control circuit (190) prohibits the first processing unit (130) from issuing load instructions or store instructions; the memory access launch control circuit (190) also prohibits the second processing unit (140) from issuing load instructions or store instructions during this cycle; during the column-level launch cycle authorized by either of the processing units, the memory access launch control circuit (190) prohibits the multiply-accumulate unit (120) from issuing load instructions or store instructions; The multiply-accumulate unit (120) performs a single multiply-accumulate operation for at least two column-level transmit cycles; during a column-level transmit cycle in which the multiply-accumulate unit (120) does not issue a load instruction or a store instruction, the memory access transmit control circuit (190) provides an authorization signal to the first processing unit (130) or the second processing unit (140) that has a memory access request.
9. A kernel scheduling method, characterized in that, Applied to the kernel architecture of any one of claims 1 to 8, the method comprises: The matrix multiplication operation of the current thread block is performed by the multiply-accumulate unit; the matrix auxiliary operation of the current thread block is performed by the first processing unit. After the matrix calculation of the current thread block is completed, the instruction decoding circuit parses the concurrent request instruction of the current thread block; the instruction decoding circuit transmits the parsing result to the request receiving interface through the scheduling request interface. The thread block selection circuit responds to the parsing result and accesses the queue interaction interface via the queue access interface to obtain the queue status; when the queue is not empty, the thread block selection circuit reads the command information of the thread block to be scheduled; the thread block selection circuit determines the next thread block based on the command information. The thread block selection circuit provides thread block scheduling information to the scheduling control interface via the distribution control interface; the instruction decoding circuit provides decoding instructions to the decoding receiving interface via the decoding sending interface. The concurrent distribution circuit distributes the matrix multiplication instruction of the next thread block to the matrix instruction receiving interface via the first instruction distribution interface, based on the thread block scheduling information; distributes the matrix auxiliary operation instruction of the next thread block to the auxiliary instruction receiving interface via the second instruction distribution interface; and distributes the post-processing instruction of the current thread block to the post-processing instruction receiving interface via the third instruction distribution interface. The matrix multiplication operation of the next thread block is performed by the multiply-accumulate unit; the matrix auxiliary operation of the next thread block is performed by the first processing unit; the post-processing operation of the current thread block is performed by the second processing unit; and the scheduler arranges the instruction distribution timing so that the matrix calculation stage of the next thread block and the post-processing stage of the current thread block overlap in execution time.
10. The kernel scheduling method according to claim 9, characterized in that, The method further includes: The selection control circuit updates the matrix calculation completion status in the scheduling status register based on the execution status received by the first execution status receiving interface and the second execution status receiving interface; the selection control circuit updates the post-processing completion status and the occupancy status of the second processing unit based on the execution status received by the third execution status receiving interface. When the post-processing stage of the current thread block is completed first, the selection control circuit updates the occupancy status of the second processing unit to idle; the multiply-accumulate unit continues to execute the matrix multiplication operation that the next thread block has not yet completed; the first processing unit continues to execute the matrix auxiliary operation that the next thread block has not yet completed. When the matrix calculation stage of the next thread block is completed first, the selection control circuit checks the occupancy status of the second processing unit; if the second processing unit is still occupied by the current thread block, the selection control circuit records the post-processing pending launch status of the next thread block. After the matrix calculation stage of the next thread block is completed, the selection control circuit sends a post-processing distribution permission to the scheduling control interface via the distribution control interface when the second processing unit is idle; the concurrent distribution circuit responds to the post-processing distribution permission and distributes the post-processing instruction of the next thread block to the post-processing instruction receiving interface via the third instruction distribution interface; the second processing unit executes the post-processing operation of the next thread block.
