Data caching method, scheduler, processor, AI chip and electronic equipment
By recording the input and output cache information and counter configuration of operators in the scoreboard, the hardware resource consumption problem of data dependency checking in AI processors is solved, thereby improving computational efficiency and resource utilization.
Patent Information
- Application Number
- CN202411156125.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2026-03-03
AI Technical Summary
In AI processors, existing technologies struggle to effectively resolve data dependencies between instructions, especially in vector processor and tensor processor architectures. Traditional scoreboard technology cannot effectively record and maintain large-granularity data cache information, leading to excessive hardware resource consumption.
By recording the input buffer information, output buffer information, and configuration information of various counters related to the output in the scoreboard, the relevant storage information can be represented in a concise and clear manner, enabling the checking and resolution of data dependencies and avoiding excessive consumption of hardware resources.
This enables a concise and clear representation of operator data dependencies within the AI processor, reducing hardware resource consumption and improving computational efficiency and resource utilization.
Smart Images
Figure CN121597276A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computers, and more specifically, to a data caching method, a scheduler, a processor, an AI chip, and an electronic device. Background Technology
[0002] In current microprocessor architectures, concurrent and out-of-order instruction execution are common practices to improve overall performance. One problem arising from concurrent and out-of-order instruction execution is the need to resolve data dependencies between instructions. However, current methods for resolving these data dependencies are not very effective. Summary of the Invention
[0003] In view of the above problems, this application proposes a data caching method, scheduler, processor, AI chip, and electronic device to solve the above problems.
[0004] In a first aspect, embodiments of this application provide a data caching method applied to a scheduler. The method includes: determining a target register corresponding to a target operator from multiple registers corresponding to the scoreboard, wherein each register in the multiple registers records configuration information of the corresponding operator, the configuration information of the operator including input cache information, output cache information, and various counters related to the output cache; calculating data cache information corresponding to the target operator based on the configuration information recorded in the target register; performing a data dependency check on the target operator based on the data cache information, and controlling whether to allow the target operator to proceed based on the data dependency check result.
[0005] Secondly, embodiments of this application provide a scheduler for executing the above-described method.
[0006] Thirdly, embodiments of this application provide a processor, which includes the aforementioned scheduler, sub-computation engine, and instruction distribution unit; the instruction distribution unit is used to read the target operator and send the target operator to the scheduler; the scheduler is used to schedule and control whether to allow the target operator; the sub-computation engine is used to process the instruction corresponding to the target operator allowed by the scheduler.
[0007] Fourthly, embodiments of this application provide an electronic device that includes the processor described above.
[0008] Fifthly, embodiments of this application provide an AI chip that includes the aforementioned processor.
[0009] The data caching method, scheduler, processor, AI chip, and electronic device provided in this application first determine the target register corresponding to the target operator from multiple registers corresponding to the scoreboard. Each register in the multiple registers records the configuration information of the corresponding operator, including the input cache information, output cache information, and various counters related to the output cache. Then, based on the configuration information recorded in the target register, the data cache information corresponding to the target operator is calculated. Finally, a data dependency check is performed on the target operator based on the data cache information, and the scheduling control determines whether to allow the target operator based on the data dependency check result. Thus, by recording the input cache information, output cache information, and various counters related to the output of the operator in the scoreboard, the relevant storage information can be clearly represented to perform data dependency checks and resolutions for the operator, avoiding excessive hardware resource consumption. Attached Figure Description
[0010] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1 A flowchart illustrating an embodiment of the instruction processing method provided in this application is shown.
[0012] Figure 2 This illustration shows a schematic diagram of the branch data cache provided in an embodiment of this application;
[0013] Figure 3 A flowchart illustrating a data caching method provided in an embodiment of this application is shown;
[0014] Figure 4 A flowchart illustrating a data caching method provided in an embodiment of this application is shown;
[0015] Figure 5 This application provides a consumer-producer schematic diagram based on an embodiment of the present application.
[0016] Figure 6 This application illustrates a scheme for calculating data volume using a counter in a scoreboard, as provided by an embodiment of the present application.
[0017] Figure 7 The topology diagram of the subgraph provided in the embodiments of this application is shown. Detailed Implementation
[0018] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.
[0019] Out-of-order processor architectures improve instruction execution efficiency and performance by allowing independent instructions to be executed out of order. However, there are often data dependencies between instructions. Therefore, a key issue that out-of-order processor architectures need to address is instruction scheduling and distribution, requiring that the execution order of instructions strictly adhere to their corresponding data dependencies.
[0020] The scoreboard technology was first proposed and implemented in the CDC 6600 and is now widely used in current architectures such as RISC-V. Previously, scoreboard technology was often used to implement hardware dynamic instruction scheduling. Scoreboard technology includes a series of registers and hardware circuitry that controls instruction issuance and scheduling. It can be understood that the scoreboard is a representation of information stored in memory. By recording the data loading status of registers on the scoreboard, it controls instruction access to registers. For example, when an instruction needs to read or write to a register, it can check the data loading status of that register recorded on the scoreboard to ensure that instruction execution does not cause data conflicts. The scoreboard first checks for structural hazards and data hazards for each instruction set during the decoding stage and generates relevant records, which are stored in registers. Then, based on these records, it determines whether the instruction can be issued for execution or needs to be paused. Simultaneously, the scoreboard monitors the availability of operands for relevant instructions to control when relevant instructions can read registers. Finally, the scoreboard controls the operation of writing output data to registers. The introduction of scoreboard technology enables out-of-order and concurrent execution of instructions, greatly improving the execution efficiency of related programs.
[0021] Currently, most AI processors employ homogeneous or heterogeneous approaches to decompose computations for parallelization. Homogeneous processing primarily uses multiple engines / sub-engines with identical functional architectures to share the overall computational task (e.g., different patches of an image). Heterogeneous processing uses multiple computational engines with different functionalities to implement different computational processing functions, such as DMA engines, scalar computation engines, vector computation engines, and matrix computation engines. This leads to the question of how to schedule these multiple engines. Existing technical solutions can be categorized based on the scheduling implementation method into hardware scheduling (dedicated schedulers, serving only specific algorithms / scenes) and software scheduling (flexible). Both scheduling modes need to address the issues of data dependencies and data hazards.
[0022] Scoreboards, as a classic hardware scheduling scheme, are widely used in various types of processors to resolve data hazards. In traditional scoreboards, the recorded information is primarily based on register dependencies.
[0023] However, AI processors are often based on vector processor architectures, or even specific tensor processor architectures such as systolic arrays. In AI processors, the amount of data required per instruction is significantly higher than that of traditional processors, often relying on vector registers (e.g., in the RV64V, a single vector register is 2048 bits wide, and there are 32 vector registers) or even memory (e.g., most systolic array input data needs to be read directly from SRAM). To resolve these data hazards, a mechanism is required to record and maintain data in input / output buffers at a larger granularity and on a wider scale (not limited to registers, but extending to other memories). Simultaneously, considering hardware area, power consumption, and complexity, the corresponding solution needs to concisely and clearly represent the relevant data storage information. The traditional approach of allocating a separate bit for each memory location, as in traditional scoreboards, is impractical.
[0024] Meanwhile, in AI neural networks, the introduction of residual topology means that for a fixed data cache, multiple operators may read data, defined as multiple consumers. Conversely, an operator may have multiple output caches, defined as multiple producers. Therefore, the scoreboard mechanism faces new requirements: data hazard detection must support a single data cache corresponding to multiple producer or consumer relationships.
[0025] To address the aforementioned problems, the inventors, through extensive research, discovered and proposed the data caching method, scheduler, processor, AI chip, and electronic device provided in this application. By recording the input cache information, output cache information, and configuration information of various counters related to the output corresponding to the operators in the scoreboard, the relevant storage information can be clearly represented to check and resolve the data dependencies of the operators, avoiding excessive consumption of hardware resources. The specific data caching method will be described in detail in subsequent embodiments.
[0026] To better understand the solutions of the embodiments of this application, the technical terms used in the embodiments of this application will be explained below.
[0027] SOC (System on Chip): System on a chip
[0028] OP (Operator): Operator
[0029] psv (preserve): Reserved
[0030] wcntr(Write Counter): Write counter
[0031] rcntr (Read Counter): Read counter
[0032] SCB (Scoreboard): Scoreboard
[0033] AI (Artificial Intelligence): Artificial Intelligence
[0034] Please see Figure 1 , Figure 1 A flowchart illustrating a data caching method according to an embodiment of this application is shown. This method records the input cache information, output cache information, and configuration information of various counters related to the output corresponding to the operators in the scoreboard. This allows for a concise and clear representation of the relevant storage information, enabling the checking and resolution of data dependencies for the operators and avoiding excessive hardware resource consumption. In a specific embodiment, this data caching method is applied to the scheduler. The following will focus on... Figure 1 The process shown will be described in detail. The data caching method may specifically include the following steps:
[0035] Step S110: Determine the target register corresponding to the target operator from the multiple registers corresponding to the scoreboard. Each of the multiple registers records the configuration information of the corresponding operator. The configuration information of the operator includes the input buffer information, output buffer information and various counters related to the output buffer.
[0036] The scoreboard technology itself can be constructed using a series of registers or other memories. In this embodiment, the scoreboard can correspond to multiple registers; that is, the scoreboard can be used to record the data loading status of multiple registers. The scoreboard can record relevant data cache information at the granularity of a single operator. Therefore, if the entire scoreboard is viewed as a table, the index of this table represents the operator identifier information (ID) of each operator, and each operator corresponds to a register recorded in the scoreboard, called an entry. It is understood that in this embodiment, each of the multiple registers corresponding to the scoreboard corresponds to one operator, and the register is used to record the configuration information of the corresponding operator.
[0037] In this embodiment, a target operator can be determined, and the register corresponding to the target operator can be selected from multiple registers corresponding to the scoreboard as the target register. The target operator can be the operator whose corresponding instruction needs to be issued to the corresponding sub-computing engine for processing.
[0038] Optionally, the target operator can be any one of the multiple operators included in the subgraph to be processed. As one approach, during the configuration phase of the subgraph to be processed, the scheduler can execute the operator configuration instructions corresponding to each of the multiple operators, allocate one register from among the multiple registers corresponding to the scoreboard for each operator, and record the configuration information of the corresponding operator in the register. Optionally, a register (containing the corresponding entry ID, where the entry ID is the operator identification information for the corresponding operator in the entire scoreboard) can be allocated to each operator according to the corresponding instruction information.
[0039] Therefore, when the target operator is determined, the target register corresponding to the target operator can be determined from the multiple registers corresponding to the scoreboard. For example, when the target operator is determined, the operator identification information of the target operator can be obtained, and the target register corresponding to the target operator can be determined from the multiple registers corresponding to the scoreboard based on the operator identification information of the target operator.
[0040] In this embodiment, each of the multiple registers records the configuration information of the corresponding operator. This operator configuration information includes input buffer information, output buffer information, and various counters related to the output buffer. Specifically, if the operator is the initial input operator in the AI network, it can be assumed that the operator has no input buffer. Therefore, the corresponding input buffer information can be left blank in the register corresponding to this operator. Similarly, if the operator has no output buffer, the corresponding output buffer information and various counters related to the output buffer can be left blank in the register corresponding to this operator.
[0041] In this embodiment, the number of input buffers corresponding to an operator can be zero, one, or more. Therefore, the scoreboard can record the number of its corresponding input buffers and the input buffer identification information (input buffer ID) used to determine each input buffer in the corresponding register. Thus, the input buffer information can include the number of input buffers and the input buffer identification information used to determine each input buffer.
[0042] Understandably, each register recorded on the scoreboard corresponds to an operator, and each operator can theoretically have an input buffer and an output buffer, which are independent of each other. For each register, this embodiment primarily aims to record information about the output buffer, including the amount of valid data and free space. However, for each operator, the information about its input buffer also needs to be maintained, as it may be necessary to check whether the amount of valid data in the input buffer meets a threshold later. Therefore, this embodiment provides an index ID (i.e., input buffer identification information) for the input buffer of the corresponding operator in one of its registers.
[0043] It should be noted that a cache may be the input cache of one or more operators, and also the output cache of one or more other operators. For example, suppose there is a first operator, a second operator, and a target cache. When the first operator executes, it needs to read data from the target cache; in this case, the target cache can be considered the input cache of the first operator. Simultaneously, when the second operator executes, it needs to write data to the target cache; in this case, the target cache can be considered the output cache of the second operator. That is, the input cache of the first operator is also the output cache of the second operator. In this embodiment, the operator identifier information of an operator can generally be set to be consistent with the output cache identifier information of its corresponding output cache. Therefore, when the input cache of the first operator serves as the output cache of the second operator, the output cache identifier information of the second operator's output cache can be the operator identifier information of the second operator. Correspondingly, the input cache identifier information of the first operator's input cache can also be the operator identifier information of the second operator.
[0044] Furthermore, because in AI networks, the input buffer of each operator is typically the output buffer of other operators, when using a specific input buffer as an output buffer, the operator identifier information of the corresponding operator can be used as the index value. This index value reveals the corresponding register, which records detailed information about the input of that operator (corresponding to the output buffers of other operators).
[0045] In this embodiment, the number of output buffers corresponding to each of the multiple operators can be zero, one, or more. Therefore, the scoreboard can record detailed information about each corresponding output buffer in its corresponding register. Furthermore, for each output buffer, the valid data in the output buffer may be consumed by one or more other operators; therefore, the detailed information for each output buffer may include, but is not limited to:
[0046] The number of consumers corresponding to each output buffer;
[0047] Total capacity of all output buffers;
[0048] The effective data volume and free space in each output buffer. The following combinations of information can provide equivalent information: cumulative consumption in the output buffer & cumulative generation in the output buffer & total capacity of the output buffer; remaining space in the output buffer & total capacity of the output buffer; effective data volume in the output buffer & total capacity of the output buffer;
[0049] When the output buffer has multiple consumers, each consumer needs to be recorded separately (e.g., the corresponding rcntr for each consumer, including the rcntr and psv_rcntr for each consumer). Because different consumers may consume data from the same output buffer at different speeds, slower-consuming consumers need to be considered when calculating free space. This prevents premature space release from causing slower-consuming consumers to be unable to obtain the correct data; for example... Figure 2 As shown, there are two different consumers: operator 1 and operator 2.
[0050] The amount of data to be consumed and generated by the issued instructions. When there are multiple consumption nodes, the amount of data to be consumed must be recorded separately for each node.
[0051] The effective data volume and free space within the output buffer can be determined by the data index rcntr that each consumer will read from the output buffer, the data index wcntr that each producer will write to the output buffer, the data index psv_rcntr that needs to be reserved in the output buffer for each consumer, and the free space psv_wcntr that needs to be reserved in the output buffer for each producer. Therefore, this output buffer information can include the number of output buffers, the total capacity of each output buffer, and the number of consumers corresponding to each output buffer. The various counters related to the output buffer may include the data index that each consumer will read from the output buffer next, the data index that each producer will write to the output buffer next, the data index that the output buffer needs to reserve for each consumer, and the free space that the output buffer needs to reserve for each producer. Optionally, the various counters related to the output may include four types of counters (first counter, second counter, third counter, and fourth counter), wherein the first counter corresponds to the data index that each consumer will read from the output buffer next, the second counter corresponds to the data index that each producer will write to the output buffer next, the third counter corresponds to the data index that the output buffer needs to reserve for each consumer, and the fourth counter corresponds to the free space that the output buffer needs to reserve for each producer.
[0052] Therefore, this embodiment uses a relatively concise counter value to calculate and determine the effective data volume and free space in the cache, and supports read and write operations by multiple consumers and multiple producers. The implementation is simple and does not consume excessive hardware resources. It is highly scalable and adaptable to different modes. Furthermore, using different types of counters can express the effective data volume, free space, and data volume that needs to be reserved in the cache, not only satisfying data dependency checks but also preventing the risk of over-issuance of the same data or space by different operators simultaneously.
[0053] Optionally, the configuration information of the operator may also include sub-engine ID, which is used to determine the sub-computing engine to which the operator is executed.
[0054] Therefore, for this target operator, its corresponding target register can record the configuration information of the target operator. The configuration information of the target operator can include the input buffer information, output buffer information and various counters related to the output buffer.
[0055] Step S120: Calculate the data cache information corresponding to the target operator based on the configuration information recorded in the target register.
[0056] In this embodiment, given a target register, the data cache information corresponding to the target operator can be calculated based on the configuration information recorded in the target register. This data cache information includes the amount of valid data in the target input cache corresponding to the target operator, and / or the free space in the target output cache corresponding to the target operator.
[0057] The configuration information recorded in the target register includes the input buffer information, output buffer information, and various counters related to the output buffer corresponding to the target operator. Based on the input buffer information, output buffer information, and various counters related to the output buffer corresponding to the target operator, the effective data volume in the target input buffer corresponding to the target operator, and / or the free space in the target output buffer corresponding to the target operator can be calculated.
[0058] In some implementations, the scheduler may have a pre-set calculation formula. When the target register is determined, the scheduler can use this formula to calculate the configuration information recorded in the target register and obtain the data cache information corresponding to the target operator.
[0059] Step S130: Perform a data dependency check on the target operator based on the data cache information, and determine whether to allow the target operator based on the data dependency check result.
[0060] In this embodiment, when the data cache information corresponding to the target operator is obtained, the data dependency relationship of the target operator can be checked according to the data cache information to obtain the data dependency relationship check result, and the scheduling control can decide whether to allow the target operator based on the data dependency relationship check result.
[0061] Optionally, if the data dependency check result determines that the target operator meets the release conditions, the target operator can be released. In this case, the instructions corresponding to the target operator can be issued to the corresponding sub-computation engine for execution. If the data dependency check result determines that the target operator does not meet the release conditions, the target operator can be temporarily denied until the data dependency check result determines that the target operator meets the release conditions. In this case, the instructions corresponding to the target operator can be issued to the corresponding sub-computation engine for execution.
[0062] It is understandable that for a target operator, if it has corresponding input data, the prerequisite for its execution includes that the corresponding operation data is ready and can be obtained. For example, assuming the target operator requires 32 bits of valid data to execute, then this input data threshold can be defined as 32 bits. That is, the amount of valid data in the target input buffer corresponding to the target operator can be defined as at least 32 bits.
[0063] It is understandable that for a target operator, if it has corresponding output data, a prerequisite for its execution includes that the storage space for the corresponding data output must have sufficient free space to support the output data of the target operator. For example, assuming the execution of the target operator will produce 64 bits of output data, then the threshold for the storage space of this output data can be defined as 64 bits. That is, it can be defined that the free space in the target output buffer corresponding to the target operator must be at least 64 bits.
[0064] In this context, it's understandable that for a target operator to be executed, if it has corresponding input and output data, the prerequisites include that the corresponding operation data is ready and can be obtained, and that the storage space for the corresponding output data has sufficient free space to support the target operator's output data. For example, assuming the target operator requires 32 bits of valid data and produces 64 bits of output data, then the input data threshold can be defined as 32 bits, and the storage space threshold for the output data can be defined as 64 bits. That is, it can be defined that the amount of valid data in the target input buffer corresponding to the target operator must be at least 32 bits, and that the free space in the target output buffer corresponding to the target operator must be at least 64 bits.
[0065] Therefore, in this embodiment, when a target operator is determined, it can be determined whether the target operator corresponds to input data and whether it corresponds to output data. Specifically, if it is determined that the target operator only corresponds to input data and not output data, the data cache information corresponding to the target operator can be defined to include the amount of valid data in the target input cache corresponding to the target operator. If it is determined that the target operator only corresponds to output data and not input data, the data cache information corresponding to the target operator can be defined to include the free space in the target output cache corresponding to the target operator. If it is determined that the target operator corresponds to both input data and output data, the data cache information corresponding to the target operator can be defined to include the amount of valid data in the target input cache corresponding to the target operator and the free space in the target output cache corresponding to the target operator.
[0066] In some implementations, the data cache information corresponding to the target operator includes the amount of valid data in the target input cache corresponding to the target operator.
[0067] Based on this, the effective data volume required for the execution of the target operator can be determined. This volume is then compared with the effective data volume in the target input buffer corresponding to the target operator to determine if the effective data volume in the target input buffer meets the required amount. If the effective data volume in the target input buffer meets the required amount, the target operator meets the release condition and can be allowed to proceed. If the effective data volume in the target input buffer does not meet the required amount, the target operator does not meet the release condition and can be temporarily withheld until the effective data volume in the target input buffer reaches the required amount, at which point the target operator can be allowed to proceed.
[0068] In some implementations, the data cache information corresponding to the target operator includes the free space in the target output cache corresponding to the target operator.
[0069] Based on this, the required free space for the target operator's output data can be determined. This free space is then compared with the free space in the target output buffer corresponding to the target operator to determine if the free space in the target output buffer meets the required free space for the target operator's output data. If the free space in the target output buffer meets the required free space for the target operator's output data, the target operator meets the release condition and can be released. If the free space in the target output buffer does not meet the required free space for the target operator's output data, the target operator does not meet the release condition and can be temporarily withheld until the free space in the target output buffer meets the required free space for the target operator's output data, at which point the target operator can be released.
[0070] In some implementations, the data cache information corresponding to the target operator includes the amount of valid data in the target input cache corresponding to the target operator and the free space in the target output cache corresponding to the target operator.
[0071] Based on this, the effective data volume required for the execution of the target operator and the free space required for the output data of the target operator can be determined. The effective data volume required for the execution of the target operator is compared with the effective data volume in the target input buffer corresponding to the target operator to determine if the effective data volume in the target input buffer meets the required effective data volume for the target operator's execution. Similarly, the free space required for the output data of the target operator is compared with the free space in the target output buffer corresponding to the target operator to determine if the free space in the target output buffer meets the required free space for the target operator's output data. If it is determined that the effective data volume in the target input buffer meets the required effective data volume for the target operator's execution, and the free space in the target output buffer meets the required free space for the target operator's output data, then the target operator meets the release conditions and can be allowed to proceed. If it is determined that the amount of valid data in the target input buffer corresponding to the target operator is less than the amount of valid data required for the execution of the target operator, or that the free space in the target output buffer corresponding to the target operator is less than the free space required for the output data of the target operator, then the target operator does not meet the release conditions. The target operator can be temporarily not released until the amount of valid data in the target input buffer corresponding to the target operator reaches the amount of valid data required for the execution of the target operator, and the free space in the target output buffer corresponding to the target operator reaches the free space required for the output data of the target operator. Only then can the target operator be released.
[0072] One embodiment of this application provides a data caching method. First, a target register corresponding to a target operator is determined from multiple registers corresponding to the scoreboard. Each register in the multiple registers records the configuration information of the corresponding operator. The configuration information of the operator includes input cache information, output cache information, and various counters related to the output cache. Then, the data cache information corresponding to the target operator is calculated based on the configuration information recorded in the target register. Finally, a data dependency check is performed on the target operator based on the data cache information, and the scheduling control determines whether to allow the target operator based on the data dependency check result. Thus, by recording the input cache information, output cache information, and various counters related to the output of the operator in the scoreboard, the relevant storage information can be clearly represented to perform data dependency checks and resolutions for the operator, avoiding excessive hardware resource consumption.
[0073] Please see Figure 3 , Figure 3 A flowchart illustrating a data caching method according to an embodiment of this application is shown. This method is applied to the aforementioned scheduler, and will be discussed below. Figure 3 The process shown will be described in detail. The data caching method may specifically include the following steps:
[0074] Step S210: Determine the target register corresponding to the target operator from the multiple registers corresponding to the scoreboard. Each of the multiple registers records the configuration information of the corresponding operator. The configuration information of the operator includes the input buffer information, output buffer information and various counters related to the output buffer.
[0075] For a detailed description of step S210, please refer to step S110, which will not be repeated here.
[0076] Step S220: Based on the operator identifier information of the target operator, find the input identifier information of the target input cache corresponding to the target operator, wherein the target input cache is another output cache corresponding to other operators, and the other operators are one or more operators other than the target operator.
[0077] In this embodiment, when a target operator is determined, the input identifier information of the target input buffer corresponding to the target operator can be found in the target register based on the operator identifier information of the target operator. It is understood that the target input buffer can serve as the output buffer for other operators; that is, the target input buffer can be defined as another output buffer corresponding to another operator. It is understood that this other operator can be one or more operators other than the target operator.
[0078] Step S230: Determine the other registers corresponding to the other operators based on the input identifier information of the target input buffer.
[0079] In this embodiment, given the input identifier information of the target input buffer, other registers corresponding to other operators can be determined based on this identifier information. It is understood that the operator identifier information of other operators can be determined based on the identifier information of the target input buffer, and the other registers corresponding to other operators can be determined based on the operator identifier information of other operators.
[0080] Step S240: Calculate the effective data volume in the target input buffer corresponding to the target operator based on the configuration information recorded in the other registers.
[0081] Optionally, the configuration information recorded in the other register may include the input buffer information, output buffer information, and various counters related to the output buffer corresponding to the other operator.
[0082] In this embodiment, when other registers corresponding to other operators are determined, the configuration information recorded in the other registers can be searched, and the effective data volume in the target input buffer corresponding to the target operator can be calculated based on the configuration information recorded in the other registers. For example, the effective data volume in the target input buffer corresponding to the target operator can be calculated based on the various counters related to the output buffer corresponding to the other operators recorded in the other registers.
[0083] As one possible approach, the configuration information recorded in the other registers includes the data indexes that other output caches need to reserve for each consumer, and the data indexes that each producer of other output caches will subsequently write into other output caches. Based on this, calculating the effective data volume in the target input cache corresponding to the target operator, according to the configuration information recorded in the other registers, can include: calculating the effective data volume in the target input cache corresponding to the target operator based on the data indexes that other output caches need to reserve for each consumer and the data indexes that each producer of other output caches will subsequently write into other output caches. For example, if the data index that other output caches need to reserve for each consumer is psv_rcntr, and the data index that each producer of other output caches will subsequently write into other output caches is wcntr, then the effective data volume in the target input cache (i.e., other output caches) is wcntr - psv_rcntr.
[0084] Step S250: Perform a data dependency check on the target operator based on the amount of valid data in the target input buffer and the amount of valid data required for the execution of the target operator, and decide whether to allow the target operator to proceed based on the data dependency check result.
[0085] For a detailed description of step S250, please refer to step S130, which will not be repeated here.
[0086] One embodiment of this application provides a data caching method, which, compared to... Figure 1 In this embodiment, the data caching method further includes searching for the input identifier information of the target input cache corresponding to the target operator based on the operator identifier information of the target operator. The target input cache is another output cache corresponding to other operators. Based on the input identifier information of the target input cache, other registers corresponding to other operators are determined. Based on the configuration information recorded in the other registers, the effective data volume in the target input cache corresponding to the target operator is calculated. Thus, the effective data volume in the target input cache corresponding to the target operator can be calculated through various counters recorded in the scoreboard-related registers to determine whether to execute the target operator, thereby enabling the checking of data dependencies.
[0087] Please see Figure 4 , Figure 4 A flowchart illustrating a data caching method according to an embodiment of this application is shown. This method is applied to the aforementioned scheduler, and will be discussed below. Figure 4 The process shown will be described in detail. The data caching method may specifically include the following steps:
[0088] Step S310: Determine the target register corresponding to the target operator from the multiple registers corresponding to the scoreboard. Each of the multiple registers records the configuration information of the corresponding operator. The configuration information of the operator includes the input buffer information, output buffer information and various counters related to the output buffer.
[0089] For a detailed description of step S310, please refer to step S110, which will not be repeated here.
[0090] Step S320: Calculate the free space in the target output buffer corresponding to the target operator based on the configuration information recorded in the target register.
[0091] In this embodiment, the free space in the target output buffer corresponding to the target operator can be calculated based on the configuration information recorded in the target register corresponding to the target operator. For example, the free space in the target output buffer corresponding to the target operator can be calculated based on the output buffer information corresponding to the target operator and various counters related to the output buffer recorded in the target register.
[0092] As one feasible approach, the configuration information recorded in the target register includes the total capacity of the target output buffer, the index of the data that each consumer will read from the target output buffer next, and the free space that the target output buffer needs to reserve for each producer. Based on this, calculating the free space in the target output buffer corresponding to the target operator, according to the configuration information recorded in the target register, can include: calculating the free space in the target output buffer corresponding to the target operator based on the total capacity of the target output buffer, the index of the data that each consumer will read from the target output buffer next, and the free space that the target output buffer needs to reserve for the producer. As an example, if the total capacity of the target output buffer is TotalSpace, the index of the data that each consumer will read from the target output buffer next is rcntr, and the free space that the target output buffer needs to reserve for the producer is psv_wcntr, then the free space in the target output buffer is Total Space - (psv_wcntr - rcntr).
[0093] Step S330: Perform a data dependency check on the target operator based on the available space in the target output buffer and the available space required for the data output by the target operator, and decide whether to allow the target operator to proceed based on the data dependency check result.
[0094] For a detailed description of step S330, please refer to step S130, which will not be repeated here.
[0095] One embodiment of this application provides a data caching method, which, compared to... Figure 1 The data caching method shown in this embodiment also calculates the free space in the target output cache corresponding to the target operator based on the configuration information recorded in the target register. Thus, the free space in the target output cache corresponding to the target operator can be calculated through various counters recorded in the scoreboard-related registers to determine whether to execute the target operator, thereby enabling the checking of data dependencies.
[0096] Therefore, it is understandable that this embodiment can use different counters to calculate and statistically analyze the effective data volume, free space, reserved data volume, and reserved free space of a single cache. To clearly illustrate the relevant technical solution, a specific example is provided below. In this example, for a cache space, it is limited to a maximum of two different operators reading data from the space, referred to as consumer0 and consumer1, and a maximum of one operator writing data to this data space, defined as the producer. Figure 5As shown, the corresponding scoreboard registers define a total of 6 different counters: rcntr0, rcntr1, wcntr, psv_rcntr0, psv_rcntr1, and psv_wcntr.
[0097] Here, rcntr0 and rcntr1 are the indices of the data that consumer0 and consumer1 will read next, respectively. This index is not necessarily equivalent to a specific address; it can be a data sequence number calculated using a custom data unit.
[0098] Here, wcntr is the index of the data that the producer will write next.
[0099] Here, psv_rcntr0 and psv_rcntr1 are the data indices that consumer0 and consumer1 need to retain, respectively. psv_rcntr-rcntr indicates that the corresponding data will be used as input data by subsequent operators. Once the relevant data is used, it will be considered invalid until the corresponding operator has finished executing, thereby preventing the same storage space data from being promised to other operators again as input data.
[0100] Here, psv_wcntr is the free space that the producer needs to reserve. psv_wcntr-wcntr means that the relevant free space has been reserved by the producer. This part of the free space is the space for storing the output data after the producer's relevant operators have been executed, and it cannot be used by other operators to write data.
[0101] Combining the above definitions, and referring to... Figure 6 As can be seen, for consumer0, the corresponding valid data is wcntr-psv_rcntr0, and for consumer1, the valid data is wcntr-psv_rcntr1. The reserved space is psv_wcntr-min(rcntr0, rcntr1), and the free space is: Total Space-(psv_wcntr-min(rcntr0, rcntr1)).
[0102] Please refer to Figure 7 And Table 1, Figure 7 Table 1, based on the functional parameter definitions of the scoreboard above, provides a specific subgraph and its register information within the scoreboard. This use case includes: a subgraph and its corresponding operator graph, the register definitions of the scoreboard, and the corresponding information of each operator within the scoreboard.
[0103] Table 1
[0104]
[0105] Among them, Figure 7 In Table 1, ATO 20 is an operator numbered 20. This operator has two input buffers: 18 and 19, and one output buffer: 20. The topology between the corresponding operators 21, 22, 25, and 26 is as follows: Figure 7 As shown in Table 1, for this specific subgraph, the configuration information recorded in the corresponding registers on the scoreboard is as follows. Taking operator 20 as an example:
[0106] arg0 describes that it has 2 inputs and 1 output.
[0107] arg1 describes the IDs of the operators corresponding to its input buffer as 18 and 19. Since a buffer may have multiple consumers, here ridx indicates that operator 20 is the 0th consumer.
[0108] arg3 describes the total capacity of the current output buffer, the number of consumers, and provides the initial values of the corresponding counters (i.e., the initial values of counters in arg4 and arg5).
[0109] arg4 and arg5 give the values of the corresponding counter.
[0110] In particular, considering that for more complex computational flow graphs, such as an operator that may correspond to multiple input buffers and multiple output buffers, the recording format of the input buffer information corresponding to the input buffers and the output buffer information corresponding to the output buffers in the scoreboard can be expanded, for example, but not limited to other formats such as linked lists.
[0111] In some embodiments, one example of this application provides a data caching device applied to the aforementioned scheduler. The data caching device includes: a target register determination module, a data caching information calculation module, and a control module, wherein:
[0112] The target register determination module is used to determine the target register corresponding to the target operator from multiple registers corresponding to the scoreboard. Each register in the multiple registers records the configuration information of the corresponding operator. The configuration information of the operator includes the input buffer information, output buffer information and various counters related to the output buffer.
[0113] The data cache information calculation module is used to calculate the data cache information corresponding to the target operator based on the configuration information recorded in the target register.
[0114] Furthermore, the data cache information calculation module includes: an input identifier information lookup submodule, an other register determination submodule, and an effective data volume calculation submodule, wherein:
[0115] The input identifier information lookup submodule is used to look up the input identifier information of the target input cache corresponding to the target operator based on the operator identifier information of the target operator, wherein the target input cache is another output cache corresponding to other operators, and the other operators are one or more operators other than the target operator.
[0116] The other register determination submodule is used to determine the other registers corresponding to the other operators based on the input identification information of the target input buffer.
[0117] The effective data volume calculation submodule is used to calculate the effective data volume in the target input buffer corresponding to the target operator based on the configuration information recorded in the other registers.
[0118] Furthermore, the configuration information recorded in the other registers includes the data indexes that the other output buffers need to retain for each consumer, and the data indexes that each producer of the other output buffers will write to the other output buffers next. The effective data volume calculation submodule includes: an effective data volume calculation unit, wherein:
[0119] The effective data volume calculation unit is used to calculate the effective data volume in the target input cache corresponding to the target operator based on the data index that the other output cache needs to retain for each consumer and the data index that each producer of the other output cache will write into the other output cache next.
[0120] Furthermore, the data cache information calculation module includes: a free space calculation submodule, wherein:
[0121] The free space calculation submodule is used to calculate the free space in the target output buffer corresponding to the target operator based on the configuration information recorded in the target register.
[0122] Furthermore, the configuration information recorded in the target register includes the total capacity of the target output buffer, the data index that each consumer will read from the target output buffer next, and the free space that the target output buffer needs to reserve for the producer. The free space calculation submodule includes: a free space calculation unit, wherein:
[0123] The spare space calculation unit is used to calculate the spare space in the target output cache corresponding to the target operator based on the total capacity of the target output cache, the data index that each consumer will read from the target output cache next, and the spare space that the target output cache needs to reserve for the producer.
[0124] The control module is used to perform data dependency checks on the target operator based on the data cache information, and to schedule and control whether to allow the target operator based on the data dependency check results.
[0125] Further, the data cache information corresponding to the target operator includes the amount of valid data in the target input cache corresponding to the target operator and the free space in the target output cache corresponding to the target operator. The control module includes: a first control submodule and a second control submodule, wherein:
[0126] The first control submodule is configured to allow the target operator to proceed if it is determined that the amount of valid data in the target input buffer corresponding to the target operator reaches the amount of valid data required for the execution of the target operator, and the free space in the target output buffer corresponding to the target operator reaches the free space required for the output data of the target operator.
[0127] The second control submodule is used to prevent the target operator from proceeding if it is determined that the amount of valid data in the target input buffer corresponding to the target operator does not reach the amount of valid data required for the execution of the target operator, or the free space in the target output buffer corresponding to the target operator does not reach the free space required for the output data of the target operator.
[0128] In some implementations, one embodiment of this application provides a scheduler for executing the above-described data caching method.
[0129] In some implementations, one embodiment of this application provides a processor that includes a scheduler for performing the data caching method described above.
[0130] Optionally, the processor may further include a sub-computation engine and an instruction distribution unit; the instruction distribution unit is used to read the target operator and send the target operator to the scheduler; the scheduler is used to schedule whether to allow the target operator; the sub-computation engine is used to process the instructions corresponding to the target operator allowed by the scheduler.
[0131] In some implementations, one embodiment of this application provides a chip that may include the processor described above.
[0132] In some embodiments, one example of this application provides an electronic device that includes the processor described above.
[0133] Optionally, the aforementioned electronic device may also include a memory, a sensor, a display, etc., without limitation.
[0134] In summary, the data caching method, scheduler, processor, AI chip, and electronic device provided in this application first determine the target register corresponding to the target operator from multiple registers corresponding to the scoreboard. Each register in the multiple registers records the configuration information of the corresponding operator, including the input cache information, output cache information, and various counters related to the output cache. Then, based on the configuration information recorded in the target register, the data cache information corresponding to the target operator is calculated. Finally, a data dependency check is performed on the target operator based on the data cache information, and the scheduling control determines whether to allow the target operator based on the data dependency check result. Thus, by recording the input cache information, output cache information, and various counters related to the output operator in the scoreboard, the relevant storage information can be clearly represented for operator data dependency checks and resolution, avoiding excessive hardware resource consumption.
[0135] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.
Claims
1. A data caching method, characterized in that, Applied to a scheduler, the method includes: From the multiple registers corresponding to the scoreboard, determine the target register corresponding to the target operator. Each of the multiple registers records the configuration information of the corresponding operator. The configuration information of the operator includes the input buffer information, output buffer information and various counters related to the output buffer. Based on the configuration information recorded in the target register, calculate the data cache information corresponding to the target operator; The target operator is checked for data dependencies based on the data cache information, and the scheduling control decides whether to allow the target operator to proceed based on the data dependency check results.
2. The method according to claim 1, characterized in that, The step of calculating the data cache information corresponding to the target operator based on the configuration information of the target operator recorded in the target register includes: Based on the operator identifier information of the target operator, the input identifier information of the target input cache corresponding to the target operator is found, wherein the target input cache is another output cache corresponding to other operators; the other operators are one or more operators other than the target operator; Based on the input identifier information of the target input buffer, determine the other registers corresponding to the other operators; Based on the configuration information recorded in the other registers, the amount of valid data in the target input buffer corresponding to the target operator is calculated.
3. The method according to claim 2, characterized in that, The configuration information recorded in the other registers includes the data indexes that the other output buffers need to retain for each consumer, and the data indexes that each producer of the other output buffers will write into the other output buffers next. The step of calculating the effective data volume in the target input buffer corresponding to the target operator based on the configuration information recorded in the other registers includes: Based on the data indexes that the other output caches need to retain for each consumer and the data indexes that each producer of the other output caches will write into the other output caches next, calculate the effective amount of data in the target input cache corresponding to the target operator.
4. The method according to claim 1, characterized in that, The step of calculating the data cache information corresponding to the target operator based on the configuration information recorded in the target register includes: Based on the configuration information recorded in the target register, calculate the free space in the target output buffer corresponding to the target operator.
5. The method according to claim 4, characterized in that, The configuration information recorded in the target register includes the total capacity of the target output buffer, the data index that each consumer will read from the target output buffer next, and the free space that the target output buffer needs to reserve for the producer. The step of calculating the free space in the target output buffer corresponding to the target operator based on the configuration information recorded in the target register includes: Based on the total capacity of the target output cache, the data index that each consumer will read from the target output cache next, and the free space that the target output cache needs to reserve for the producer, calculate the free space in the target output cache corresponding to the target operator.
6. The method according to any one of claims 1-5, characterized in that, in: The input cache information includes the number of input caches and input identifier information used to determine each input cache; The output cache information includes the number of output caches, the total capacity of each output cache, and the number of consumers corresponding to each output cache. The various counters related to the output cache include the data index that each consumer will read from the output cache next, the data index that the producer will write to the output cache next, the data index that needs to be reserved in the output cache for each consumer, and the spare space that the output cache needs to reserve for the producer.
7. The method according to any one of claims 1-5, characterized in that, The configuration information of the operator also includes engine identification information corresponding to the operator, wherein the engine identification information is used to determine the sub-computation engine used to execute the operator.
8. The method according to any one of claims 1-5, characterized in that, The data cache information corresponding to the target operator includes the amount of valid data in the target input cache corresponding to the target operator and the free space in the target output cache corresponding to the target operator. The step of performing a data dependency check on the target operator based on the data cache information, and then controlling whether to allow the target operator based on the data dependency check result, includes: If it is determined that the amount of valid data in the target input buffer corresponding to the target operator reaches the amount of valid data required for the execution of the target operator, and the free space in the target output buffer corresponding to the target operator reaches the free space required for the output data of the target operator, then the target operator is allowed to proceed; or If it is determined that the amount of valid data in the target input buffer corresponding to the target operator is less than the amount of valid data required for the execution of the target operator, or the free space in the target output buffer corresponding to the target operator is less than the free space required for the output data of the target operator, then the target operator will not be allowed to proceed.
9. A scheduler, characterized in that, The scheduler is used to execute the method as described in any one of claims 1-8.
10. A processor, characterized in that, The processor includes the scheduler, sub-computation engine, and instruction dispatch unit as described in claim 9; The instruction distribution unit is used to read the target operator and send the target operator to the scheduler; The scheduler is used to control whether to allow the target operator to proceed. The sub-computing engine is used to process the instructions corresponding to the target operator that the scheduler allows.
11. An AI chip or electronic device, characterized in that, Includes the processor as described in claim 10.