A Hybrid Scheduling System and Method for Asynchronous Prefetching of High-Priority Activation Values and Dual Queues
By employing a high-priority activation value asynchronous prefetching and a dual-queue hybrid scheduling system in the neural network accelerator, the prefetching commands for weights and activation values are stored independently and executed asynchronously. This solves the pipeline stall problem caused by delayed activation value scheduling after weight preloading, improves the time overlap between data preparation and computation, and reduces memory access idle time.
Patent Information
- Application Number
- CN202611609934.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-10-19
- Publication Date
- 2026-11-17
AI Technical Summary
When performing operators such as matrix multiplication and convolution on neural network accelerators or graphics processors, the weight preloading task may delay the scheduling of activation values, causing the computational core pipeline to stall. How can we reduce scheduling blockage when weight preloading and activation value prefetching are concurrent?
A high-priority activation value asynchronous prefetching and dual-queue hybrid scheduling system is adopted. It is equipped with a computing core sequencer, a prefetch sequencer, a secondary cache and a data prefetching engine. It utilizes the physical independence of the dynamic prefetch queue and the static preload queue, combined with a priority arbitrator, to asynchronously perform activation value prefetching and computing core instruction execution. The two types of commands are stored independently, and the commands of the dynamic prefetch queue are scheduled first.
It effectively reduces the scheduling blockage of activation value acquisition caused by weight preloading, reduces the idle time of the computing pipeline caused by memory access, improves the time overlap between data preparation and computing, and reduces pipeline pauses.
Smart Images

Figure CN123114779A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of chip architecture and processor memory access scheduling technology, and in particular to a high-priority activation value asynchronous prefetching and dual-queue hybrid scheduling system and method. Background Technology
[0002] When neural network accelerators or graphics processing units (GPUs) perform operators such as matrix multiplication and convolution, they need to retrieve weights and activation values from external memory and utilize on-chip caches to shorten data access latency. Weights are model parameters, which can usually be determined before the computation kernel starts, while activation values are intermediate feature data for neural network operations, and their use is related to the computation progress.
[0003] When two types of data share a command queue and are processed in a predetermined order, large batches of preloaded weight tasks that enter the queue earlier may delay the scheduling of activation values. When activation values are on the critical path of computation, this waiting can cause pipeline stalls in the computation core.
[0004] Therefore, how to reduce the scheduling blockage of activation value acquisition caused by weight preloading when two types of memory access tasks are concurrent is a technical problem that needs to be solved. Summary of the Invention
[0005] This application proposes a hybrid scheduling system and method for asynchronous prefetching of high-priority activation values and dual queues, aiming to reduce scheduling blockage when weight preloading and activation value prefetching occur concurrently, and to reduce pipeline pauses caused by the computation core waiting for activation values. This helps to shorten the scheduling wait time for activation values and reduce idle time in the computation pipeline caused by memory access.
[0006] In a first aspect, embodiments of this application provide a high-priority activation value asynchronous prefetching and dual-queue hybrid scheduling system, including a computing core equipped with a computing core sequencer, a prefetch sequencer, a secondary cache, and a data prefetch engine. The data prefetch engine includes a dynamic prefetch queue, a static preload queue, and a priority arbitrator. The dynamic prefetch queue is connected to the computing core sequencer, and the static preload queue is connected to the prefetch sequencer. The dynamic prefetch queue and the static preload queue are physically independent and both are connected to the priority arbitrator. The data prefetch engine is connected to an external memory and the secondary cache, and the secondary cache is connected to the computing core. The dynamic prefetch queue is used to cache the activation value prefetch commands issued by the computing core sequencer during the operation of the computing kernel. The activation values are intermediate feature data of the neural network operation. The static preload queue is used to cache the weight preload commands issued by the prefetch sequencer; The priority arbiter is used to prioritize the scheduling of commands in the dynamic prefetch queue, and only schedule commands in the static preload queue when the queue is empty. The data prefetching engine is used to prefetch corresponding data from the external memory to the secondary cache according to the scheduled command. The prefetching of the activation value is performed asynchronously with the instruction execution of the computing core.
[0007] As can be seen, in this embodiment, the two physically independent queues allow the weight preloading command and the activation value prefetch command to have independent storage locations for scheduling. The priority arbitrator ensures that the command scheduler will prioritize serving the activation value when the dynamic prefetch queue is not empty. Since the activation value prefetching and the instruction execution of the computation core are asynchronous, data transfer can overlap with the instruction execution that does not depend on the prefetch result in time. This helps to reduce queuing interference caused by weight preloading, shorten the scheduling wait of activation values, and reduce pipeline stalls.
[0008] In one possible example, the computational core sequencer is connected to the dynamic prefetch queue via a non-blocking bus interface, and the dynamic prefetch queue and the static preload queue use independent hardware storage units; The computational core sequencer is used to issue a prefetch command for the activation value of the current thread block to the dynamic prefetch queue at the beginning of the current thread block. The computing core is used to execute subsequent instructions that do not depend on the prefetch result of the activation value prefetch command during the execution of the activation value prefetch command.
[0009] As can be seen in this example, independent hardware storage units separate the queue capacities for the two types of commands, and the non-blocking bus interface separates the submission of prefetch commands from the return of prefetch results. The computation core processes subsequent instructions without data dependencies during command execution, which is beneficial for utilizing computational resources during memory access wait times.
[0010] In one possible example, the computational core sequencer is used to determine a look-ahead thread block based on the grid coordinates of the current thread block and a preset grid step size, wherein the look-ahead thread block is a thread block to be executed located after the current thread block along the preset grid dimension; The computational core sequencer is also used to issue an activation value prefetch command for the lookahead thread block to the dynamic prefetch queue before the matrix operation instruction of the current thread block begins to be executed. The data prefetching engine is used to prefetch the activation values required by the lookahead thread block during the matrix operation performed by the current thread block.
[0011] As can be seen in this example, the lookahead thread block is determined by the execution position of the current thread block, and its prefetch command is issued before the current matrix operation begins. Therefore, the computation period of the current thread block can be used to transfer the activation values of subsequent thread blocks. Combined with the priority scheduling of the dynamic prefetch queue, this helps reduce data waiting time after thread block switching.
[0012] In one possible example, the preset grid dimension is the row dimension of the output grid, and the preset grid step size represents the output row span covered by the computing core in one computation cycle; The computational core sequencer is used to determine the grid coordinates of the look-ahead thread block through hardware logic; The grid coordinates of the look-ahead thread block in the row dimension are the coordinates of the current thread block in the row dimension after being offset by the preset grid step size in an increasing direction. The grid coordinates of the look-ahead thread block in the other grid dimensions are the same as those of the current thread block.
[0013] As can be seen, in this example, the hardware logic determines the subsequent prefetch position according to the span of the output row covered by one calculation loop, so that the look-ahead range matches the execution direction of the thread block. This helps to reduce the control overhead in address preparation and reduce the possibility of misalignment between the prefetched object and the subsequent calculation requirements.
[0014] In one possible example, the data prefetch engine includes a tensor mapping parser that is connected to the dynamic prefetch queue; The activation value prefetch command is associated with an activation value tensor descriptor, which is used to characterize the tensor base address, the shape of the data block to be prefetched, the storage step size of each dimension, and the grid coordinates of the corresponding thread block. The storage step size characterizes the address interval between adjacent elements along the corresponding dimension. The tensor mapping resolver is used to determine the set of prefetch addresses of the corresponding activation values in the external memory based on the activation value tensor descriptor. The data prefetching engine is used to prefetch the activation values required by the current thread block or the lookahead thread block based on the prefetch address set.
[0015] As can be seen in this example, the tensor map resolver converts the logical data range corresponding to the thread block into an access address in external memory, enabling the prefetching of the current thread block and the lookahead thread block to adapt to multidimensional tensors and cross-row storage layout, which helps to improve the consistency between prefetched data and computational requirements.
[0016] In one possible example, the secondary cache includes a weighted persistent cache and an activation value regular cache; The prefetch sequencer is used to issue the weight preload command before the computing kernel starts; The data prefetching engine is used to prefetch the weights corresponding to the weight preload command into the weight persistent cache area; The data prefetching engine is also used to prefetch the activation value corresponding to the activation value prefetching command into the activation value regular cache area; The weight persistence cache is used to retain weights that can be repeatedly accessed by the computing kernel during the operation of the computing kernel; The activation value cache is used to temporarily store activation values for the computing core to read.
[0017] As can be seen, in this example, the combination of preloading and continuous retention of weights, and the combination of runtime prefetching and temporary storage of activation values, allows the preparation timing and caching methods of the two types of data to adapt to their respective reuse requirements. The dual-queue scheduling helps to balance the timeliness of activation value acquisition with the efficiency of repeated access to weights.
[0018] In one possible example, the number of computing cores is at least two, and at least two computing cores are connected to the same L2 cache, with the computing core sequencer in each computing core connected to the dynamic prefetch queue. The computing core sequencer of each computing core is used to concurrently issue activation value prefetch commands for their respective first batch of thread blocks to the dynamic prefetch queue at the initial stage of the computing core startup. The first batch of thread blocks are located in different rows or columns of the output grid. The data prefetching engine is used to fill the second-level cache with the corresponding activation value according to the activation value prefetching command for the first batch of thread blocks; The computing core is used to read the activation value from the secondary cache when the activation value required by the first batch of thread blocks is reused in subsequent thread blocks and the secondary cache is hit.
[0019] As can be seen in this example, multiple computing cores fill activation values into a shared, accessible L2 cache during the initial startup phase, allowing subsequent thread blocks to reuse the already filled and still valid data. Under conditions of data reuse and cache hits, this helps reduce repeated external reads of the same activation value.
[0020] In one possible example, the data prefetch engine is connected to the computational core sequencer; The data prefetching engine is used to return a prefetching completion notification to the computing core sequencer after the activation value corresponding to the activation value prefetching command is written to the second-level cache. The prefetching completion notification indicates that the cache filling corresponding to the activation value prefetching command has been completed. The computational core sequencer is used to confirm the completion status of the activation value prefetch command based on the prefetch completion notification.
[0021] As can be seen in this example, the prefetch completion notification is returned after the corresponding cache is filled, enabling the computation core sequencer to distinguish between the two states of command submission and data filling. This is beneficial for confirming the data preparation progress and maintaining the data dependencies of subsequent instructions during asynchronous prefetching.
[0022] Secondly, embodiments of this application provide a high-priority activation value asynchronous prefetching and dual-queue hybrid scheduling method applied to a data prefetching engine. The data prefetching engine includes a dynamic prefetch queue, a static preload queue, and a priority arbitrator. The dynamic prefetch queue and the static preload queue are physically independent and both connected to the priority arbitrator. The dynamic prefetch queue is connected to a computing core sequencer within the computing core, and the static preload queue is connected to the prefetch sequencer. The data prefetching engine is connected to external memory and a secondary cache, and the secondary cache is connected to the computing core. The method includes: The weight preload command issued by the prefetch sequencer is cached in the static preload queue; The activation value prefetch command issued by the computing core sequencer during the operation of the computing kernel is cached in the dynamic prefetch queue, and the activation value is the intermediate feature data of the neural network operation; The priority arbitrator determines the command to be executed based on the non-empty state of the dynamic prefetch queue. When the dynamic prefetch queue is not empty, the command to be executed is taken from the dynamic prefetch queue. When the dynamic prefetch queue is empty, the command to be executed is taken from the static preload queue. According to the command to be executed, the corresponding data is prefetched from the external memory to the secondary cache, and the prefetching of the activation value is asynchronous with the instruction execution of the computing core.
[0023] As can be seen, in this embodiment, the two types of commands enter physically independent queues, and the state of the dynamic prefetch queue determines the source of the current command scheduling. This method combines priority scheduling of activation values with asynchronous transport, which helps to reduce the queuing interference of weight preloading on the computation critical path and provides conditions for the time overlap of memory access and instruction execution.
[0024] In one possible example, the computational core sequencer is connected to the dynamic prefetch queue via a non-blocking bus interface; the activation value prefetch commands are for the current thread block and the lookahead thread block, respectively, where the lookahead thread block is a subsequent thread block to be executed, determined by the computational core sequencer based on the grid coordinates of the current thread block and a preset grid step size; the activation value prefetch commands for both the current thread block and the lookahead thread block are written to the dynamic prefetch queue via the non-blocking bus interface before the matrix operation instructions of the current thread block begin execution; the method further includes... During the matrix operation performed by the current thread block, the activation values required by the lookahead thread block are prefetched into the second-level cache according to the activation value prefetch command for the lookahead thread block.
[0025] As can be seen, in this example, the commands of the current thread block and the lookahead thread block are submitted in advance through the non-blocking bus interface. During the current matrix operation, the prefetching of subsequent thread blocks can be performed, which helps to move the preparation process of subsequent data forward and reduce the memory access gap when the thread blocks are executed continuously.
[0026] In one possible example, the activation value prefetch command is associated with an activation value tensor descriptor, which characterizes the tensor base address, the shape of the data block to be prefetched, the storage stride in each dimension, and the grid coordinates of the corresponding thread block. The storage stride characterizes the address interval between adjacent elements along the corresponding dimension. The method further includes... The prefetch address set of the corresponding activation value in the external memory is determined based on the activation value tensor descriptor; The activation value required for the current thread block or the lookahead thread block is prefetched from the external memory according to the prefetch address set.
[0027] As can be seen, in this example, the prefetch address is determined by the storage layout in the tensor descriptor and the position of the thread block, which enables lookahead prefetching to be implemented at the corresponding external storage address. This is beneficial for adapting to non-contiguous tensor storage layouts and reducing the movement of irrelevant data.
[0028] In one possible example, the number of computing cores is at least two, and at least two computing cores are connected to the same L2 cache. The computing core sequencer within each computing core is connected to the dynamic prefetch queue. The method also includes... In the early stage of the startup of the computing kernel, the activation value prefetch commands issued concurrently by the computing kernel sequencer of each computing kernel are cached in the dynamic prefetch queue. The activation value prefetch commands are respectively for the first batch of thread blocks located in different rows or columns of the output grid. The activation value is filled into the second-level cache according to the activation value prefetch command for the first batch of thread blocks. The activation value is used by subsequent thread blocks to read from the second-level cache when the required activation value is the same and the second-level cache is hit.
[0029] As can be seen in this example, the multi-core startup prefetch prepares reusable activation values for the shared L2 cache. Subsequent thread blocks can directly read the data when the data is the same and the cache is hit, which helps to extend the startup filling benefits to subsequent concurrent computations.
[0030] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to be executed by the processor, the programs including instructions for performing the steps in the first aspect of embodiments of this application.
[0031] Fourthly, embodiments of this application provide a computer storage medium storing a computer program for electronic data interchange, wherein the computer program causes a computer to perform some or all of the steps described in the first aspect of this embodiment.
[0032] As can be seen, in this embodiment, compared to the method of placing weight preloading commands and activation value prefetching commands in the same command queue for unified scheduling, this application addresses the differences in the pre-preparability and computational timeliness requirements of the two types of data, reducing the queuing interference of weight preloading on activation value prefetching. The scheme in this application uses a dynamic prefetch queue to cache activation value prefetching commands issued by the computation core sequencer, and a static preloading queue to cache weight preloading commands issued by the prefetch sequencer. A priority arbitrator limits the command scheduling source when the dynamic prefetch queue is not empty, and the corresponding data is prefetched to the connected secondary cache through the data prefetching engine. The activation value prefetching process is asynchronous with the instruction execution of the computation core. Because the queue storage of the two types of commands is independent, the weight preloading command that arrives first will not occupy the queue position of the activation value command; because the dynamic prefetch queue only schedules commands when it is not empty, the weight command will not continue to occupy the command scheduling opportunity. Combined with asynchronous prefetching, activation value transport can overlap with the execution of instructions that do not depend on the prefetch result in time, thereby helping to shorten the scheduling wait of activation values and reduce the idle time of the computation pipeline caused by memory access. Attached Figure Description
[0033] Figure 1 This is a schematic diagram of the structure of the high-priority activation value asynchronous prefetching and dual-queue hybrid scheduling system provided in the embodiments of this application.
[0034] Figure 2 This is a timing diagram illustrating the non-blocking command submission and independent instruction execution provided in an embodiment of this application.
[0035] Figure 3 This is a schematic diagram illustrating the overlapping timing of the current thread block calculation and the lookahead thread block activation value prefetching, provided as an embodiment of this application.
[0036] Figure 4 This is a schematic diagram illustrating the coordinate relationship of determining the lookahead thread block along the output grid row dimension, as provided in an embodiment of this application.
[0037] Figure 5 This is a schematic diagram illustrating the mapping relationship between the activation value tensor descriptor and the prefetch address set provided in the embodiments of this application.
[0038] Figure 6 This is a schematic diagram of the weight and activation value prefetch path and secondary cache area configuration provided in the embodiments of this application.
[0039] Figure 7This is a schematic diagram illustrating the common filling and reuse of the L2 cache by multiple computing cores, as provided in an embodiment of this application.
[0040] Figure 8 This is a timing diagram illustrating the return of cache filling completion and prefetch completion notifications provided in an embodiment of this application.
[0041] Figure 9 This is a flowchart illustrating the asynchronous prefetching of high-priority activation values and the dual-queue hybrid scheduling method provided in this application embodiment.
[0042] Figure 10 This is a flowchart illustrating the tensor address resolution and activation value prefetching method provided in an embodiment of this application.
[0043] Figure 11 A flowchart illustrating the multi-computing core startup activation value filling method provided in this application embodiment. Detailed Implementation
[0044] The following embodiments illustrate this application in conjunction with system architecture, command scheduling, and data transfer processes. System embodiments use uniform component labeling, while method embodiments only number the steps. The specific thread block locations and data sizes in each embodiment are used to explain the processing procedure and do not constitute a limitation on the applicable data scale.
[0045] See Figure 1 , Figure 1 This is a schematic diagram of the structure of the high-priority activation value asynchronous prefetching and dual-queue hybrid scheduling system provided in the embodiments of this application.
[0046] The system includes a computing core 110 with a computing core sequencer 111, a prefetch sequencer 120, a secondary cache 130, and a data prefetch engine 140. The data prefetch engine 140 includes a dynamic prefetch queue 141, a static preload queue 142, and a priority arbitrator 143. The dynamic prefetch queue 141 is connected to the computing core sequencer 111, and the static preload queue 142 is connected to the prefetch sequencer 120. The dynamic prefetch queue 141 and the static preload queue 142 are physically independent and both are connected to the priority arbitrator 143. The data prefetch engine 140 is connected to an external memory 150 and the secondary cache 130, and the secondary cache 130 is connected to the computing core 110.
[0047] The computing core 110 is a hardware processing unit that executes neural network operator instructions, the computing core sequencer 111 is a control circuit that controls the order in which instructions are issued by the hardware processing unit, and the prefetch sequencer 120 is a control circuit that issues early memory access commands. The computing kernel is the instruction program that completes the specified operator operation, and is distinct from the computing core 110 that carries the program.
[0048] The L2 cache 130 is a data storage component located in the processor cache hierarchy, and the data prefetch engine 140 is a control component that initiates data transfer transactions; the two cooperate via a data transfer connection. The external memory 150 can be a double data rate (DFR) memory or a high-bandwidth memory. In the schematic diagram, the computing core sequencer 111 is located inside the computing core 110, and the two queues and priority arbitrator 143 are located inside the data prefetch engine 140. The L2 cache 130 and the data prefetch engine 140 are represented as independent components.
[0049] The two queues are physically independent, meaning that each type of command to be scheduled has its own hardware storage location and enqueue / dequeue status. One type of command occupies the space of its own queue without consuming storage space in the other queue. The two queues can share subsequent data transfer resources; the physical independence of the queues does not require that the access channels of the external memory 150 also be independent.
[0050] The dynamic prefetch queue 141 is used to cache the activation value prefetch commands issued by the computing core sequencer 111 during the operation of the computing kernel. The activation values are intermediate feature data for neural network operations.
[0051] The activation value prefetch command is a control information request to read activation values in advance. This control information is cached in the queue, and the corresponding activation value data is carried by L2 cache 130. For example, the feature matrix output by the previous neural network layer can be used as the activation value to be prefetched in the next layer's computation. The issuance of dynamic prefetch representation commands is related to the progress of the computation kernel, rather than representing a dynamic change in the queue's hardware capacity.
[0052] The static preload queue 142 is used to cache the weight preload commands issued by the prefetch sequencer 120; Weights are the model parameters used by the operator, while the weight preload command is a request for control information to prepare weights before use. Static preload indicates that the weights to be prepared can usually be determined in advance, rather than implying that the weights are unchangeable in all computational tasks. The independent sources of these two commands allow computation progress-related requests to directly enter their corresponding scheduling positions.
[0053] The priority arbiter 143 is used to prioritize the scheduling of commands from the dynamic prefetch queue 141, and only schedule commands from the static preload queue 142 when the queue is empty.
[0054] Priority scheduling indicates that at the arbitration point where the conditions for issuing a command are met, the active value is selected first to prefetch the command. When the dynamic prefetch queue 141 is not empty, the priority arbitrator 143 selects a command from this queue. When the dynamic prefetch queue 141 is empty and the static preload queue 142 is not empty, the command is selected from the static preload queue 142. When both queues are empty, no new prefetch command is issued.
[0055] For example, if the static preload queue 142 already has three weight preload commands, and the dynamic prefetch queue 141 receives an activation value prefetch command, the priority arbitrator 143 will select this activation value prefetch command during the next available command scheduling opportunity. Weight preload commands that have not yet been issued will remain in their original queues and will only have a scheduling opportunity when the dynamic prefetch queue 141 is empty.
[0056] To address the potential confusion between scheduling blocking and transmission blocking during command switching, this embodiment limits the priority selection boundary to the command scheduling stage. Transactions already sent to external memory 150 can continue to complete, and prioritizing newly arrived activation value commands does not require the cancellation of in-transit transactions. This maintains the execution continuity of issued transactions while reducing the occupation of subsequent scheduling opportunities by unissued weight commands.
[0057] The data prefetch engine 140 is used to initiate a prefetch access to the external memory 150 according to the scheduled command, receive prefetched data from the external memory 150, and prefetch the corresponding data into the L2 cache 130. The prefetching of the activation value is asynchronous with the instruction execution of the computing core 110.
[0058] The completion of asynchronous prefetch transactions is not a prerequisite for computation core 110 to continue executing all subsequent instructions. After the prefetch command is submitted, the data prefetch engine 140 handles memory access, and computation core 110 can advance instructions that do not depend on the prefetch result. Operations that actually use the activation value still require operand availability.
[0059] As can be seen, this embodiment eliminates the mutual occupation caused by two types of commands sharing the same queue position by using independent command storage, limits the interference of scheduled weight commands on activation values by using priority arbitration, and provides a time window for overlapping computation and data transfer through asynchronous execution. These three factors work together on the three interconnected stages of command entry, command selection, and data preparation, which helps reduce the waiting time after the activation value enters the critical computation path.
[0060] See Figure 2 , Figure 2 This is a timing diagram illustrating the non-blocking command submission and independent instruction execution provided in an embodiment of this application.
[0061] The computing core sequencer 111 submits an activation value prefetch command via the non-blocking bus interface 112 and sends a prefetch request to the data prefetch engine 140. The computing core sequencer 111 is connected to the dynamic prefetch queue 141 via the non-blocking bus interface 112. The dynamic prefetch queue 141 and the static preload queue 142 use independent hardware storage units.
[0062] The non-blocking bus interface 112 is a hardware interface that allows command submission and transaction completion to occur at different times. After the interface accepts a command, the command issuer does not need to wait for all the data corresponding to the command to be returned before it can continue processing instructions that can be executed independently. The independent hardware storage units can use their own register storage arrays or on-chip storage arrays to hold the command entries of the two queues.
[0063] The computational core sequencer 111 is used to issue a prefetch command for the activation value of the current thread block to the dynamic prefetch queue 141 at the beginning of the current thread block.
[0064] A thread block is a group of threads that jointly undertake a task of calculating a block of output data. The current thread block is the thread block that computing core 110 is currently preparing to execute or is currently executing. The initial stage represents the preparation period after the thread block obtains its execution position and before the relevant computing body unfolds. At this time, activation value prefetching commands can be submitted according to its data requirements.
[0065] The computing core 110 is used to execute subsequent instructions that do not depend on the prefetch result of the activation value prefetch command during the execution of the activation value prefetch command.
[0066] Subsequent instructions that do not depend on the prefetch result are those whose required operands and control conditions do not need to be provided by the current activation value prefetch. For example, subsequent address preparation, loop control, or computation using ready data can continue to be executed. In the timing diagram, the execution interval of the above instructions after the command submission ends is asynchronously overlapped with the background memory access interval, and the operations that depend on the activation value are located at the position where the data availability condition is met.
[0067] To address the issue of confusion between interface backpressure and data waiting, non-blocking does not require accepting new commands when the queue is full. When there are no free entries in the queue, the interface can wait for available entries. Once a command is accepted, its completion wait is separated from the independent instruction execution of the computing core 110. This maintains queue capacity constraints while preserving the ability of asynchronous prefetching to cover memory access latency.
[0068] As can be seen, this embodiment decouples command submission and data return on the core side based on dual-queue priority scheduling. Independent queues provide a clear command acceptance location, and the non-blocking interface and independent instruction execution together reduce unnecessary pauses caused by prefetched transactions not yet being completed.
[0069] See Figure 3 , Figure 3 This is a schematic diagram illustrating the overlapping timing of the current thread block calculation and the lookahead thread block activation value prefetching, provided as an embodiment of this application.
[0070] The computational core sequencer 111 is used to determine a lookahead thread block based on the grid coordinates of the current thread block and a preset grid step size. The lookahead thread block is a thread block to be executed located after the current thread block along a preset grid dimension.
[0071] Here, grid coordinates serve as the position identifier of the thread block within the operator's output task distribution. The preset grid dimension represents the direction of advancement for selecting subsequent tasks, and the preset grid step size is the position parameter used to determine the prefetch interval along that direction. The lookahead thread block is determined by its current execution position and the aforementioned parameters, rather than solely by inferring future addresses from past data accesses.
[0072] For example, in a two-dimensional output grid, if the current thread block is located in a row of a column, the computational core sequencer 111 keeps the column position unchanged and determines the position of the subsequent row along a preset direction, using the thread block to be executed at that position as the lookahead thread block. The prefetch object is thus associated with the computational task of the subsequent output block.
[0073] The computational core sequencer 111 is also used to issue an activation value prefetch command for the lookahead thread block to the dynamic prefetch queue 141 before the matrix operation instruction of the current thread block begins to be executed.
[0074] In a specific execution sequence, the computation core sequencer 111 submits the current thread block activation value prefetch command, and then submits the lookahead thread block activation value prefetch command. Both submissions occur before the matrix operation instructions for the current thread block begin execution. When the data required by the current thread block is available, the computation core 110 begins the matrix operation for the current thread block, and the data prefetch engine 140 performs the lookahead thread block activation value prefetch. This current thread block matrix operation and the lookahead thread block activation value prefetch satisfy the overlap between computation and lookahead prefetch. When the lookahead activation value is available, the computation core 110 executes the lookahead thread block operation. The lookahead thread block commands can then be processed by the data prefetch engine 140.
[0075] The data prefetching engine 140 is used to prefetch the activation values required by the lookahead thread block during the matrix operation performed by the current thread block.
[0076] The timing diagram shows that the current matrix operation interval and the lookahead data transport interval overlap. When the time window provided by the current matrix operation is sufficient to cover the remaining transport time of the lookahead data, subsequent thread blocks can obtain the filled activation value when using the data. If the time window is insufficient, there may still be remaining waiting.
[0077] To address the issue that subsequent thread blocks may not exist at the end of the output grid, the look-ahead position should fall within the valid grid range of the current computation task. In practice, the validity of this position can be verified before issuing the command, confirming that the corresponding activation value belongs to data that is ready to be read. This constraint is used to avoid issuing invalid prefetches for non-existent tasks or data that has not yet been read, ensuring that the data preparation for the look-ahead is consistent with the actual computational needs.
[0078] As can be seen, this embodiment transforms the current thread block's execution progress into a preparation time for subsequent activation values. Combined with the non-blocking command submission and the priority scheduling of the dynamic prefetch queue 141, it helps to bring subsequent data transfer into the current computation period in advance and reduce memory access waits at thread block junctions.
[0079] See Figure 4 , Figure 4 This is a schematic diagram illustrating the coordinate relationship of determining the lookahead thread block along the output grid row dimension, as provided in an embodiment of this application.
[0080] The preset grid dimension is the row dimension of the output grid, and the preset grid step size represents the output row span covered by the computing core 110 in one calculation cycle.
[0081] The output grid is the logical coordinate space for organizing thread block tasks according to the output data blocks. The row dimension of the output grid is the coordinate direction corresponding to the advancement of the output row, and the row dimension has an increasing direction. The output row span represents the range covered in that direction after the computing core 110 completes one computing cycle. The preset grid step size and grid coordinates use the same unit of measurement.
[0082] The computational core sequencer 111 is used to determine the grid coordinates of the lookahead thread block through hardware logic.
[0083] The hardware logic is a combinational circuit or timing circuit that implements coordinate transformation in the computational core sequencer 111. For example, the coordinates of the current row direction and the configured preset grid step size can be sent to the addition logic, and the coordinates of the remaining grid dimensions can be directly passed to the address preparation process of the lookahead command, with the coordinates of the remaining grid dimensions being the same.
[0084] The grid coordinates of the look-ahead thread block in the row dimension are the coordinates of the current thread block in the row dimension after being offset by the preset grid step size in an increasing direction. The grid coordinates of the look-ahead thread block in the other grid dimensions are the same as those of the current thread block.
[0085] For example, using the position of the starting row of the output block as the row direction coordinate, if the starting row of the current thread block is row 32 (illustrated as "Current thread block starting row: 32"), and a calculation loop covers 16 rows, offsetting 16 rows along the row dimension, then the starting row of the lookahead thread block is row 48 (illustrated as "Lookahead thread block starting row: 48"), and its column direction and batch direction positions remain unchanged. The row numbers mentioned above start from zero and are only used to illustrate the coordinate progression relationship. If the specific hardware uses thread block numbers to represent grid positions, the output row span should be converted to the coordinate units used by that grid.
[0086] To address the issue of confusion between logical grid step size and physical storage step size, a preset grid step size controls the position of subsequent tasks, while the storage step size controls the address interval between data elements; these are quantities at different levels. First, determine the lookahead thread block, and then map its corresponding data range to storage addresses, thus avoiding directly treating the output row span as a byte address offset.
[0087] As can be seen, this embodiment, based on the overlap between look-ahead prefetching and current computation, ensures that the hardware coordinate advancement is consistent with the output range covered by one computation cycle. This helps to reduce the control operations required for look-ahead address preparation and maintain the data demand connection between adjacent computation cycles.
[0088] See Figure 5 , Figure 5 This is a schematic diagram illustrating the mapping relationship between the activation value tensor descriptor and the prefetch address set provided in the embodiments of this application.
[0089] The data prefetching engine 140 includes a tensor mapping parser 144, which is connected to the dynamic prefetching queue 141.
[0090] The tensor map resolver 144 is a hardware parsing circuit that performs the translation of multidimensional data layout to memory address. Its connection with the dynamic prefetch queue 141 is used to pass descriptive information associated with the scheduled command. This parsing circuit is located inside the data prefetch engine 140, and the subsequent use of the prefetch address is still constrained by the command selection of the priority arbitrator 143.
[0091] The activation value prefetch command is associated with an activation value tensor descriptor, which is used to characterize the tensor base address, the shape of the data block to be prefetched, the storage step size of each dimension, and the grid coordinates of the corresponding thread block. The storage step size characterizes the address interval between adjacent elements along the corresponding dimension.
[0092] A tensor is a data set organized according to multiple dimensions. The tensor base address is the starting address of this data set in the storage space, and the shape of the data block to be prefetched is the number of elements in each dimension. The activation value tensor descriptor is a structured metadata expressing the above layout and thread block position. It can be passed with the command or pointed to by an index in the command. When using index association, the parsing process uses the description content corresponding to the command.
[0093] Storage stride is used to describe address changes when a multidimensional logical location changes. For example, in a two-dimensional tensor, the address interval between adjacent elements in the same row can be equal to the number of bytes occupied by one element, while the address interval between corresponding elements in two adjacent rows can be greater than the number of bytes occupied by a row of valid data due to row padding. The grid coordinates of the corresponding thread block are used to determine the position of the local data block needed in this instance within the tensor.
[0094] The tensor mapping resolver 144 is used to determine the set of prefetch addresses of the corresponding activation value in the external memory 150 based on the activation value tensor descriptor.
[0095] The prefetch address set includes address range one, address range two, and address range three, and accesses the corresponding storage locations. The tensor mapping resolver 144 can first determine the starting logical location of the local data block, then form an offset relative to the tensor base address according to the storage step size of each dimension, and finally determine the elements or continuous address ranges to be covered based on the shape of the data block. The prefetch address set represents the storage locations to be accessed in this data transfer, which can be represented as several continuous ranges or as scattered address entries.
[0096] For example, a two-dimensional activation tensor has 8 valid elements per row, each element occupying 2 bytes, and the address interval between corresponding elements in adjacent rows is 32 bytes. Starting from zero, if the data block to be prefetched starts from row 1, column 2, and has a shape of 2 rows and 3 columns, then the first row of data starts 36 bytes after the tensor's base address, the second row starts 68 bytes after the base address, and 6 bytes are read from each row. The above example illustrates that even with end-of-row padding, local data blocks can still be accurately located.
[0097] The data prefetch engine 140 is used to prefetch the activation values required by the current thread block or the lookahead thread block according to the prefetch address set, and to receive activation value data from the external memory 150.
[0098] The current thread block and the lookahead thread block can use the same tensor base address, data block shape, and storage stride, but use their respective grid coordinates to determine different sets of prefetch addresses. When adjacent thread blocks partially overlap in the data they require, the actual cache access is still handled according to the address and data validity status; different grid positions do not necessarily indicate completely different required activation values.
[0099] To address the issue of logical data blocks spanning multiple physical address segments, contiguous addresses can be grouped into ranges suitable for storage access, while non-contiguous portions are preserved as separate ranges. This allows prefetch requests to cover the required data, reducing the likelihood of interline padding or other irrelevant data being mistakenly treated as activation values. Specific transactions can be implemented at the cache line granularity; a cache line is the data unit for cache allocation and transfer.
[0100] As can be seen, this embodiment connects the determination of the look-ahead position with the parsing of the tensor storage layout, enabling the computational requirements carried by high-priority commands to be converted into executable memory access ranges, which helps to improve the accuracy of current prefetching and look-ahead prefetching under different tensor layouts.
[0101] See Figure 6 , Figure 6 This is a schematic diagram of the weight and activation value prefetch path and secondary cache area configuration provided in the embodiments of this application.
[0102] The secondary cache 130 includes a weighted persistent cache area 131 and an activation value regular cache area 132.
[0103] The weight persistence cache 131 is retained and repeatedly accessed during operation, serving as a cache storage area for reusing weights; the activation value regular cache 132 temporarily stores and reads on demand, serving as a cache storage area for intermediate feature data required during operation. The region division can be reflected in the distinction between cache capacity or cache allocation location; both regions belong to the second-level cache 130.
[0104] The prefetch sequencer 120 is used to issue the weight preload command before the computing kernel starts.
[0105] The prefetch engine 140 calculates the weights issued before kernel startup during the preparation period before the computation instructions for the operator have begun running. The prefetch sequencer 120 can issue weight preload commands after the weight addresses and data sizes have been determined. The commands are issued before startup, but it is not required that all weight transfers be completed before startup; unscheduled weight preload commands can continue to be stored in the static preload queue 142.
[0106] The data prefetching engine 140 is used to prefetch the weights corresponding to the weight preload command into the weight persistence cache 131.
[0107] The weight preloading path associates the scheduling commands obtained from the static preloading queue 142 with the weight persistence buffer 131; the data prefetch engine 140 accesses external memory, and the data in the external memory 150 returns the weight data. If an activation value prefetch command appears after the computation kernel starts, the scheduling weight command yields its scheduling opportunity according to the aforementioned arbitration rules, and the scheduling opportunity is obtained again when the dynamic prefetch queue 141 is empty.
[0108] The data prefetch engine 140 is also used to prefetch the activation value corresponding to the activation value prefetch command into the activation value regular cache area 132.
[0109] The activation value prefetch path associates the scheduled commands obtained from the dynamic prefetch queue 141 with the activation value regular cache 132. The data prefetch engine 140 calculates the activation values issued during kernel runtime. The data prefetch engine 140 determines the cache filling area according to the data category corresponding to the command, so that the runtime preparation of the activation value and the advance preparation of the weight are coordinated at the destination storage location.
[0110] The weight persistence cache 131 is used to retain weights that can be repeatedly accessed by the computing core 110 during the operation of the computing kernel.
[0111] Persistent representation maintains the weights cached during the relevant use of the computation kernel, allowing for repeated access across multiple computation steps. This does not require the weights to be permanently retained after the computation task ends. In practical implementations, the range of data to be persistently retained can be selected by combining the weight working set with the capacity of the configuration region; the working set is the set of data that needs to be repeatedly accessed during the relevant computation.
[0112] The activation value regular cache 132 is used to temporarily store activation values for the computing core 110 to read; the weights and activation values in the secondary cache 130 are read by the computing core 110.
[0113] The temporary representation activation value occupies cache space during the current computation and related reuse, and can be updated as needed. In the example combined with look-ahead prefetching, to address the issue that premature prefetching may prolong cache occupancy, the look-ahead distance is linked to the progress of adjacent computation cycles. This ensures that data enters the cache close to its usage time, reducing capacity pressure caused by premature resident data.
[0114] As can be seen, this embodiment coordinates the priority of command selection with the time requirement for data residency. Preparing and reserving reusable weights in advance creates conditions for scheduling activation values during runtime, while temporarily storing activation values as needed facilitates the turnover of cache space, thereby balancing the access efficiency of both types of data.
[0115] See Figure 7 , Figure 7This is a schematic diagram illustrating the common filling and reuse of the L2 cache by multiple computing cores, as provided in an embodiment of this application.
[0116] The number of computing cores 110 is at least two, and at least two computing cores 110 are connected to the same secondary cache 130. The computing core sequencer 111 in each computing core 110 is connected to the dynamic prefetch queue 141.
[0117] In this context, multiple computing cores 110 share a secondary cache 130, meaning that each computing core 110 can read valid data from the same cache storage space through its own access path. The secondary cache 130 contains cache line 1, cache line 2, cache line 3, ..., cache line N. For example, if two computing cores 110 execute different thread blocks, their activation value prefetch requests are sent to the same dynamic prefetch queue 141, and the returned data enters the shared secondary cache 130.
[0118] The computing core sequencer 111 of each computing core 110 is used to concurrently issue activation value prefetch commands for their respective first batch of thread blocks at the initial stage of the computing core startup, wherein the first batch of thread blocks outputs different rows or columns of the grid.
[0119] The initial startup phase of the computing kernel represents the stage where the first batch of thread blocks begins to build the required cached data. The first batch of thread blocks consists of one or more thread blocks initially undertaken by each computing core 110 during this phase. Concurrent issuance indicates that the command generation process of multiple cores can overlap in time, and it is not required that all commands be written to the same queue port in the same clock cycle.
[0120] Command acceptance required by multiple computing cores 110 can be arranged according to the write capability of the interface, and all accepted activation value prefetch commands enter the dynamic prefetch queue 141. This maintains both the concurrent relationship of requests initiated by multiple cores and the priority scheduling relationship between the dynamic prefetch queue 141 and the static preload queue 142.
[0121] The data prefetch engine 140 is used to fill the corresponding activation value into the secondary cache 130 according to the activation value prefetch command for the first batch of thread blocks.
[0122] Startup-phase population transforms the data requirements of the initial thread blocks into cached content accessible to multiple compute cores 110. When the initial thread blocks are located in different rows, activation value blocks corresponding to those rows can be prepared first. Subsequent thread blocks expanded along other output columns then use the prepared data. The specific reuse direction depends on the data access relationships of the operators.
[0123] The computing core 110 is used to read the activation value from the second-level cache 130 when the subsequent thread block reuses the activation value required by the first batch of thread blocks and meets the conditions that the required activation value is the same and the cached data is valid.
[0124] For example, in matrix multiplication, a set of activation value blocks corresponding to a set of output rows can be used in conjunction with weight blocks corresponding to different output columns for calculation. The two computation cores 110 first process the first batch of thread blocks at different row positions, writing the two sets of activation values into the L2 cache 130. When processing thread blocks of other output columns at these row positions, the same activation values can be reused.
[0125] To address the issue of assuming data reusability solely based on similar thread block positions or shapes, this embodiment presupposes that the actual required activation values are the same. Cache hits also require that the data at the corresponding address is still effectively stored in the L2 cache 130. Therefore, startup-time filling provides the conditions for subsequent hits, and the actual benefit depends on the operator's reuse relationship, cache capacity, and intermediate access patterns.
[0126] As can be seen, this embodiment links high-priority prefetching initiated by the core side with space reuse in the shared cache, enabling a single data preparation during startup to serve multiple subsequent thread blocks. Provided the corresponding data is still hit, this helps reduce redundant external reads and related data transfer overhead.
[0127] See Figure 8 , Figure 8 This is a timing diagram illustrating the return of cache filling completion and prefetch completion notifications provided in an embodiment of this application.
[0128] The data prefetch engine 140 is connected to the computing core sequencer 111.
[0129] The connection between the data prefetch engine 140 and the computational core sequencer 111 carries the prefetch completion notification. This notification, together with the previous command submission, forms an asynchronous transaction request and completion relationship. The connection can be implemented via existing communication interfaces, without requiring additional independent data storage components.
[0130] The data prefetch engine 140 is used to execute prefetch transactions. After the activation value corresponding to the activation value prefetch command is written to the second-level cache 130, it returns a prefetch completion notification to the computing core sequencer 111. The prefetch completion notification indicates that the cache filling corresponding to the activation value prefetch command has been completed.
[0131] A prefetch completion notification is a completion event indicating that the cache filling required by the corresponding command has been completed. A command reception acknowledgment is not the same as a prefetch completion notification. For commands involving multiple address ranges, a notification should be generated after the filling range corresponding to the command is completed. The reception acknowledgment of the command entering the queue cannot be used as the completion notification.
[0132] The computational core sequencer 111 is used to confirm the completion status of the command based on the prefetch completion notification.
[0133] In the example combining lookahead prefetching, after the computation core sequencer 111 submits an activation value prefetch command, it continues to advance the current thread block. After the data prefetch engine 140 completes writing the corresponding data into the activation value, it returns a notification, which the computation core sequencer 111 uses to confirm that the command has been completed. When multiple prefetch commands are not completed simultaneously, the implementation can use command identifiers passed with the request and notification to ensure that each notification corresponds one-to-one with the original command.
[0134] To address the potential confusion between "fill completion" and "continued availability," a completion notification indicates that the corresponding cache has been filled when the notification is generated, but it does not mean that the data will not be replaced later. Compute core 110 still processes data according to the cache hit and valid status during actual reads. This ensures accurate recording of asynchronous transaction progress while maintaining the correctness of normal cache accesses.
[0135] As can be seen, this embodiment supplements the non-blocking command submission with completion feedback, making the three stages of submission, execution and completion distinguishable, which is beneficial for organizing subsequent data dependencies, while retaining the ability to execute independent instructions during the waiting period.
[0136] See Figure 9 , Figure 9 This is a flowchart illustrating the asynchronous prefetching of high-priority activation values and the dual-queue hybrid scheduling method provided in this application embodiment.
[0137] The method is applied to a data prefetching engine, which includes a dynamic prefetch queue, a static preload queue, and a priority arbitrator. The dynamic prefetch queue 141 and the static preload queue 142 are physically independent and both connected to the priority arbitrator. The dynamic prefetch queue is connected to the computing core sequencer within the computing core, and the static preload queue is connected to the prefetch sequencer. The data prefetching engine is connected to external memory and a secondary cache, and the secondary cache is connected to the computing core. The method includes the following steps: Step S101: Cache the weight preload command issued by the prefetch sequencer into the static preload queue; The data prefetching engine is the main execution component of this method. Two queues carry commands yet to be scheduled, a priority arbitrator determines the source of the current command, and a secondary cache carries the data returned from prefetching. The dynamic prefetch queue 141 and the static preload queue 142 are physically independent, allowing different types of commands to enter their respective hardware storage locations. The computing core uses the prepared data via the secondary cache.
[0138] The weight preload command, originating from the prefetch sequencer, represents a request for advance preparation of the model parameters required by the operator. This step establishes the weight command's pending scheduling state, and its execution does not require the computational core to simultaneously issue activation value prefetch commands.
[0139] Step S102: The activation value prefetch command issued by the computing core sequencer during the operation of the computing kernel is cached in the dynamic prefetch queue, where the activation value is the intermediate feature data of the neural network operation. The activation value prefetch command is issued by the computational core sequencer in conjunction with the execution requirements of the computational kernel. The required data can be the intermediate feature blocks used by the current operator. The weight command enqueue and the activation value command enqueue can occur at different times, or they can be processed concurrently by their respective interfaces. The step numbering does not require that the two types of commands have a fixed arrival order.
[0140] Step S103: The priority arbitrator determines the command to be executed based on the non-empty state of the dynamic prefetch queue. The command to be executed when the dynamic prefetch queue is not empty is taken from the dynamic prefetch queue, and the command to be executed when the dynamic prefetch queue is empty is taken from the static preload queue. A non-empty state indicates that there are valid commands waiting to be scheduled in the dynamic prefetch queue. Each time the conditions for command scheduling are met, the priority arbitrator checks this state. If it is non-empty, the active value command is selected for prefetching; if it is empty, commands in the static preload queue are allowed to be selected. If both queues are empty, the queue remains in a waiting state.
[0141] For cases where earlier commands already exist in the static preload queue, the command arrival time does not alter the priority relationship between the queues. Commands awaiting scheduling in the static preload queue are retained until a low-priority scheduling opportunity becomes available, while transactions that have left the queue and are being sent to external memory are completed according to the existing memory access process. This process aligns the queue selection rules with the completion rules for transactions in transit.
[0142] Step S104: According to the command to be executed, the corresponding data is prefetched from the external memory to the secondary cache. The prefetching of the activation value is asynchronous with the instruction execution of the computing core.
[0143] The corresponding data is determined by the access range specified by the command to be executed, and is prefetched into the L2 cache for subsequent reading by the compute core. During the activation value transfer, the compute core can execute instructions that do not depend on the prefetch result; however, the data availability condition must still be met when the activation value is actually used. After completing one command selection, the queue status can be checked again at a subsequent scheduler opportunity, without limiting the two types of enqueueing processes to only one execution.
[0144] As can be seen, this embodiment utilizes independent queuing to maintain the storage boundaries of the two types of commands, uses dynamic prefetch queue state to achieve priority selection of activation values, and provides the possibility of overlapping memory access and execution through asynchronous transport, which is beneficial to shorten the scheduling wait of activation value commands in the weight preloading concurrent scenario.
[0145] The computational core sequencer is connected to the dynamic prefetch queue via a non-blocking bus interface; the activation value prefetch commands are respectively for the current thread block and the lookahead thread block, where the lookahead thread block is the subsequent thread block to be executed determined by the computational core sequencer based on the grid coordinates and preset grid step size of the current thread block; the activation value prefetch commands for the current thread block and the lookahead thread block are written to the dynamic prefetch queue via the non-blocking bus interface before the matrix operation instructions of the current thread block begin execution; the method further includes the following steps: Step S201: During the matrix operation performed by the current thread block, the activation value required by the lookahead thread block is prefetched into the second-level cache according to the activation value prefetch command for the lookahead thread block.
[0146] The non-blocking bus interface separates the data return waiting after a command is accepted from the independent instruction execution of the computation core. The current thread block is the currently assigned output block task, while the lookahead thread block is the subsequent task determined according to its grid position and preset grid step size. Both types of commands are written to a dynamic prefetch queue before the current matrix operation begins, allowing commands to participate in priority scheduling before the corresponding data is actually used.
[0147] For example, the computational core sequencer submits two commands at the beginning of the current thread block: one for the current thread block and one for a subsequent thread block determined along a preset direction. After the operands required by the current thread block meet the execution conditions, matrix operations begin. During this operation, the data prefetch engine moves the activation values of the subsequent thread blocks. This step in the process falls within the time interval of the current matrix operation, and the two operations overlap.
[0148] To address the issue of invalid data transfer due to inconsistencies between the prospective target and the actual subsequent task, the preset grid step size can be aligned with the computational progress range of the current task. Furthermore, the prospective position is confirmed to be within the valid task range before issuing commands. This process ensures that the pre-transferred data has a clearly defined subsequent user, thus reducing computational wait times caused by prefetching.
[0149] As can be seen, this embodiment inherits the priority scheduling relationship of the dynamic prefetch queue and arranges the preparation of activation values for subsequent thread blocks during the current calculation period. When the provided overlap time is sufficient to cover the corresponding transport delay, subsequent thread blocks can utilize the prepared data, and when time is insufficient, it also helps to reduce residual waiting.
[0150] See Figure 10 , Figure 10 This is a flowchart illustrating the tensor address resolution and activation value prefetching method provided in an embodiment of this application.
[0151] The activation value prefetch command is associated with an activation value tensor descriptor, which is used to characterize the tensor base address, the shape of the data block to be prefetched, the storage step size of each dimension, and the grid coordinates of the corresponding thread block. The storage step size characterizes the address interval between adjacent elements along the corresponding dimension. The method further includes the following steps: Step S301: Determine the set of prefetch addresses for the corresponding activation value in the external memory based on the activation value tensor descriptor; Among them, the activation value tensor descriptor is a structured information that expresses the multidimensional data storage layout and local data location, the tensor base address provides the address starting point, the data block shape limits the scope of this access, the storage step size describes how positional changes in different dimensions are converted into address changes, and the grid coordinates of the thread block identify the computational task to which the access scope belongs.
[0152] When determining the prefetch address set, the position of the thread block can be first converted into the starting logical position of the corresponding activation value block, and then the storage location of each element or continuous data segment within the data block can be determined according to the address intervals of each dimension. The preset grid step size is used to select the lookahead task, and the storage step size in the descriptor is used to determine the data address; the two are not interchangeable.
[0153] Step S302: Prefetch the activation value required for the current thread block or the lookahead thread block from the external memory according to the prefetch address set.
[0154] For example, for two-dimensional activation values with padding between rows, the first row's prefetch interval is determined first, and then the intervals for subsequent rows are determined according to the storage step size in the row direction. The read range of each row is limited by the shape of the data block. The current thread block and the lookahead thread block can share tensor layout information and form corresponding address sets through their respective grid positions.
[0155] To address the potential confusion between different thread block locations when reusing description information, the thread block location in the command and its associated tensor description information should both participate in the address determination. This allows for the reuse of stable layout information while ensuring that different commands point to their respective required data blocks, maintaining the accuracy of lookahead prefetching.
[0156] As can be seen, this embodiment connects non-blocking lookahead commands, tensor address resolution, and external data transfer into a complete data preparation process, enabling high-priority prefetching to adapt to multi-dimensional and non-contiguous storage layouts, which helps reduce irrelevant data readings and improves the data availability for subsequent calculations.
[0157] See Figure 11 , Figure 11 A flowchart illustrating the multi-computing core startup activation value filling method provided in this application embodiment.
[0158] The number of computing cores is at least two, and at least two computing cores are connected to the same secondary cache. The computing core sequencer within each computing core is connected to the dynamic prefetch queue. The method further includes the following steps: Step S401: In the initial stage of the computing kernel startup, the activation value prefetch commands issued concurrently by the computing kernel sequencer of each computing kernel are cached in the dynamic prefetch queue. The activation value prefetch commands are respectively for the first batch of thread blocks located in different rows or columns of the output grid. In this system, multiple computing cores are connected to the same L2 cache, allowing activation values initiated and filled by one core to be accessed by other cores. Each computing core sequencer is connected to a dynamic prefetch queue, ensuring that activation value prefetching requests from multiple cores are subject to an activation value-first queue scheduling rule.
[0159] The initial startup phase involves establishing input data for the first batch of thread blocks. These initial thread blocks are located at different positions on the output grid, corresponding to the same or different activation value ranges. Concurrent issuance indicates that each core can generate prefetch requests within overlapping time intervals, and the specific queuing process can be adapted to the receiving capacity of the queue interface.
[0160] Step S402: Fill the second-level cache with the corresponding activation value according to the activation value prefetch command for the first batch of thread blocks. The activation value is used by subsequent thread blocks to read from the second-level cache when the required activation value is the same and the second-level cache is hit.
[0161] For example, two compute cores first process thread blocks with different output rows. The data prefetching engine fills the corresponding row range with activation values into the second-level cache. Then, the two compute cores process other output columns within these row ranges. If subsequent thread blocks use the same activation values and the cache is hit, the existing cached content can be read, reducing redundant external data fetching.
[0162] Regarding the conditional relationship between startup-phase filling and subsequent reuse, this embodiment uses identical data and effective cache hits as prerequisites for reuse. The initial thread blocks being in different rows or columns is merely a task distribution method and cannot prove that their data is necessarily available for use by any subsequent thread block. The actual reuse scope should be determined by the data access relationships of the operators.
[0163] As can be seen, this embodiment connects command concurrency during multi-core startup, priority scheduling of dynamic prefetch queues, and subsequent reuse of the secondary cache. Under the condition that there are the same data requirements and the cache remains valid, it is beneficial to reduce the external memory access overhead of subsequent concurrent computing.
[0164] This application also provides a computer storage medium storing a computer program for electronic data interchange, which causes a computer to perform some or all of the steps of any of the methods described in the above method embodiments, wherein the computer includes an electronic device.
[0165] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.
[0166] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.
[0167] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical or other forms.
[0168] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0169] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0170] If the aforementioned integrated units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.
[0171] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk, etc.
[0172] The embodiments of this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A hybrid scheduling system for asynchronous prefetching of high-priority activation values and dual queues, characterized in that, The system includes a computing core equipped with a computing core sequencer, a prefetch sequencer, a secondary cache, and a data prefetch engine. The data prefetch engine includes a dynamic prefetch queue, a static preload queue, and a priority arbitrator. The dynamic prefetch queue is connected to the computing core sequencer, and the static preload queue is connected to the prefetch sequencer. The dynamic prefetch queue and the static preload queue are physically independent and both are connected to the priority arbitrator. The data prefetch engine is connected to external memory and the secondary cache, and the secondary cache is connected to the computing core. The dynamic prefetch queue is used to cache the activation value prefetch commands issued by the computing core sequencer during the operation of the computing kernel. The activation values are intermediate feature data of the neural network operation. The static preload queue is used to cache the weight preload commands issued by the prefetch sequencer; The priority arbiter is used to prioritize the scheduling of commands in the dynamic prefetch queue, and only schedule commands in the static preload queue when the queue is empty. The data prefetching engine is used to prefetch corresponding data from the external memory to the secondary cache according to the scheduled command. The prefetching of the activation value is performed asynchronously with the instruction execution of the computing core.
2. The system according to claim 1, characterized in that, The computing core sequencer is connected to the dynamic prefetch queue via a non-blocking bus interface. The dynamic prefetch queue and the static preload queue use independent hardware storage units. The computational core sequencer is used to issue a prefetch command for the activation value of the current thread block to the dynamic prefetch queue at the beginning of the current thread block. The computing core is used to execute subsequent instructions that do not depend on the prefetch result of the activation value prefetch command during the execution of the activation value prefetch command.
3. The system according to claim 2, characterized in that, The computational core sequencer is used to determine a look-ahead thread block based on the grid coordinates of the current thread block and a preset grid step size. The look-ahead thread block is a thread block to be executed located after the current thread block along the preset grid dimension. The computational core sequencer is also used to issue an activation value prefetch command for the lookahead thread block to the dynamic prefetch queue before the matrix operation instruction of the current thread block begins to be executed. The data prefetching engine is used to prefetch the activation values required by the lookahead thread block during the matrix operation performed by the current thread block.
4. The system according to claim 3, characterized in that, The preset grid dimension is the row dimension of the output grid, and the preset grid step size represents the output row span covered by the computing core in one calculation cycle; The computational core sequencer is used to determine the grid coordinates of the look-ahead thread block through hardware logic; The grid coordinates of the look-ahead thread block in the row dimension are the coordinates of the current thread block in the row dimension after being offset by the preset grid step size in an increasing direction. The grid coordinates of the look-ahead thread block in the other grid dimensions are the same as those of the current thread block.
5. The system according to claim 3, characterized in that, The data prefetching engine includes a tensor mapping parser, which is connected to the dynamic prefetching queue. The activation value prefetch command is associated with an activation value tensor descriptor, which is used to characterize the tensor base address, the shape of the data block to be prefetched, the storage step size of each dimension, and the grid coordinates of the corresponding thread block. The storage step size characterizes the address interval between adjacent elements along the corresponding dimension. The tensor mapping resolver is used to determine the set of prefetch addresses of the corresponding activation values in the external memory based on the activation value tensor descriptor. The data prefetching engine is used to prefetch the activation values required by the current thread block or the lookahead thread block based on the prefetch address set.
6. The system according to any one of claims 1-5, characterized in that, The secondary cache includes a weighted persistent cache area and an activation value regular cache area; The prefetch sequencer is used to issue the weight preload command before the computing kernel starts; The data prefetching engine is used to prefetch the weights corresponding to the weight preload command into the weight persistent cache area; The data prefetching engine is also used to prefetch the activation value corresponding to the activation value prefetching command into the activation value regular cache area; The weight persistence cache is used to retain weights that can be repeatedly accessed by the computing kernel during the operation of the computing kernel; The activation value cache is used to temporarily store activation values for the computing core to read.
7. The system according to any one of claims 1-5, characterized in that, The number of computing cores is at least two, and at least two computing cores are connected to the same secondary cache. The computing core sequencer in each computing core is connected to the dynamic prefetch queue. The computing core sequencer of each computing core is used to concurrently issue activation value prefetch commands for their respective first batch of thread blocks to the dynamic prefetch queue at the initial stage of the computing core startup. The first batch of thread blocks are located in different rows or columns of the output grid. The data prefetching engine is used to fill the second-level cache with the corresponding activation value according to the activation value prefetching command for the first batch of thread blocks; The computing core is used to read the activation value from the secondary cache when the activation value required by the first batch of thread blocks is reused in subsequent thread blocks and the secondary cache is hit.
8. The system according to any one of claims 2-5, characterized in that, The data prefetching engine is connected to the computing core sequencer; The data prefetching engine is used to return a prefetching completion notification to the computing core sequencer after the activation value corresponding to the activation value prefetching command is written to the second-level cache. The prefetching completion notification indicates that the cache filling corresponding to the activation value prefetching command has been completed. The computational core sequencer is used to confirm the completion status of the activation value prefetch command based on the prefetch completion notification.
9. A hybrid scheduling method for asynchronous prefetching of high-priority activation values and dual queues, characterized in that, This is applied to a data prefetching engine, which includes a dynamic prefetch queue, a static preload queue, and a priority arbitrator. The dynamic prefetch queue and the static preload queue are physically independent and both are connected to the priority arbitrator. The dynamic prefetch queue is connected to the computing core sequencer within the computing core, and the static preload queue is connected to the prefetch sequencer. The data prefetching engine is connected to external memory and a secondary cache, and the secondary cache is connected to the computing core. The method includes: The weight preload command issued by the prefetch sequencer is cached in the static preload queue; The activation value prefetch command issued by the computing core sequencer during the operation of the computing kernel is cached in the dynamic prefetch queue, and the activation value is the intermediate feature data of the neural network operation; The priority arbitrator determines the command to be executed based on the non-empty state of the dynamic prefetch queue. When the dynamic prefetch queue is not empty, the command to be executed is taken from the dynamic prefetch queue. When the dynamic prefetch queue is empty, the command to be executed is taken from the static preload queue. According to the command to be executed, the corresponding data is prefetched from the external memory to the secondary cache, and the prefetching of the activation value is asynchronous with the instruction execution of the computing core.
10. The method according to claim 9, characterized in that, The computational core sequencer is connected to the dynamic prefetch queue via a non-blocking bus interface; the activation value prefetch commands are respectively for the current thread block and the lookahead thread block, where the lookahead thread block is a subsequent thread block to be executed, determined by the computational core sequencer based on the grid coordinates of the current thread block and a preset grid step size; the activation value prefetch commands for both the current thread block and the lookahead thread block are written to the dynamic prefetch queue via the non-blocking bus interface before the matrix operation instructions of the current thread block begin execution; the method further includes... During the matrix operation performed by the current thread block, the activation values required by the lookahead thread block are prefetched into the second-level cache according to the activation value prefetch command for the lookahead thread block.
11. The method according to claim 10, characterized in that, The activation value prefetch command is associated with an activation value tensor descriptor, which is used to characterize the tensor base address, the shape of the data block to be prefetched, the storage stride of each dimension, and the grid coordinates of the corresponding thread block. The storage stride characterizes the address interval between adjacent elements along the corresponding dimension. The method also includes... The prefetch address set of the corresponding activation value in the external memory is determined based on the activation value tensor descriptor; The activation value required for the current thread block or the lookahead thread block is prefetched from the external memory according to the prefetch address set.
12. The method according to claim 9, characterized in that, The number of computing cores is at least two, and at least two computing cores are connected to the same secondary cache. The computing core sequencer within each computing core is connected to the dynamic prefetch queue. The method also includes... In the early stage of the startup of the computing kernel, the activation value prefetch commands issued concurrently by the computing kernel sequencer of each computing kernel are cached in the dynamic prefetch queue. The activation value prefetch commands are respectively for the first batch of thread blocks located in different rows or columns of the output grid. The activation value is filled into the second-level cache according to the activation value prefetch command for the first batch of thread blocks. The activation value is used by subsequent thread blocks to read from the second-level cache when the required activation value is the same and the second-level cache is hit.
