Artificial intelligence chips, methods for executing host-issued requests, computing devices, media, and program products.

Abstract

This invention relates to an artificial intelligence chip, a method for executing requests issued by a host, a computing device, a computer-readable storage medium, and a computer program product. The artificial intelligence chip includes: a computing core configured to execute requests issued by a host; a target storage unit configured to store updated data associated with the request; and multiple levels of bypass units disposed in an on-chip region adjacent to the computing core. The multiple levels of bypass units are configured to: back up updated data associated with a previous request to a predetermined level of bypass unit; and, in response to determining that the current request to be executed has a data dependency with a previous request, retrieve the backed-up updated data of the previous request from the corresponding level of the bypass unit of the previous request with the data dependency, for use in executing the current request. This invention enables uninterrupted operation of the request processing pipeline when faced with a data dependency between the current request and a previous request, significantly improving the chip's processing efficiency.

Description

Technical Field

[0001] The embodiments of the present invention generally relate to the field of artificial intelligence technology, and more specifically to an artificial intelligence chip, a method for executing host-issued requests, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] The efficiency of request processing in an artificial intelligence (AI) chip (such as, but not limited to, general-purpose graphics processing units) determines its performance. AI chips typically employ a pipelined architecture, and ensuring that processing units can uninterruptedly process requests issued by the host (e.g., the CPU) is a core element in improving chip performance. It should be understood that traditional AI chips incorporate multi-level cache systems. For example, the cache system within a computing core typically uses Static Random Access Memory (SRAM) as its primary storage medium. Taking the L1 cache as an example, the data within the computing core is mainly stored in the L1 cache. The computing core needs to continuously read data from SRAM and write new data back to SRAM after several clock cycles.

[0003] In traditional methods of executing host-issued requests, when data dependencies exist between adjacent pipeline requests, subsequent requests cannot obtain the latest data from preceding requests, necessitating pipeline interruption. Subsequent requests must wait for the preceding request to complete before processing. For example, if there is a data dependency between request 0 and request 1, such as both reading data from address A, request 1 needs to access the data at address A in Static Random Access Memory (SRAM) at clock cycles N (where N is a natural number). Since request 0 updates the data at address A in SRAM at clock cycles N+2, request 1 cannot obtain the latest data at address A in SRAM before clock cycles N+2. It must wait for request 0 to complete writing to SRAM before continuing to execute request 1, causing pipeline interruption and hindering uninterrupted operation. Therefore, traditional methods significantly impact chip processing efficiency when facing data dependencies between requests.

[0004] In summary, the shortcomings of traditional AI chips and methods for executing host-issued requests are that when there are data dependencies between requests, the request processing pipeline cannot operate continuously, thus affecting the chip's processing efficiency. Summary of the Invention

[0005] This invention provides an artificial intelligence chip, a method for executing host-issued requests, a computing device, a computer-readable storage medium, and a computer program product, which enable the request processing pipeline to operate uninterrupted even when there is a data dependency between the current request and previous requests, significantly improving the chip's processing efficiency.

[0006] According to a first aspect of the present invention, an artificial intelligence chip is provided. The artificial intelligence chip includes: an artificial intelligence chip comprising: a computing core configured to execute a request issued by a host; a target storage unit configured to store update data associated with the request; and multiple levels of bypass units disposed in an on-chip region adjacent to the computing core, the multiple levels of bypass units being configured to: back up update data associated with a previous request to a bypass unit of a predetermined level in the multiple levels of bypass units; and, in response to determining that a current request to be executed has a data dependency with a previous request, to retrieve the backed-up update data of the previous request from the bypass unit of the corresponding level of the previous request with the data dependency, for use in executing the current request.

[0007] According to a second aspect of the present invention, a method for executing a host-issued request is also provided, the method comprising: storing update data associated with a previous request issued by the host to a target storage unit, and simultaneously backing up the update data associated with the previous request to a predetermined level of a multi-level bypass unit, the multi-level bypass units being configured in an on-chip region adjacent to a computing core; determining whether there is a data dependency between the current request to be executed and the previous request; and in response to determining that there is a data dependency between the current request to be executed and the previous request, obtaining the backed-up update data of the previous request from the bypass unit of the corresponding level of the previous request with the data dependency, for use in executing the current request.

[0008] According to a third aspect of the invention, a computing device is also provided. The computing device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the computing device to perform the method of the second aspect of the invention.

[0009] According to a fourth aspect of the present invention, a computer-readable storage medium is also provided. The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method of the second aspect of the present invention.

[0010] According to a fifth aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a machine, performs the method of the second aspect of the present invention.

[0011] In some embodiments, the number of levels of the multi-level bypass units is equal to the difference between the number of clock cycles the computing core uses to read and update data for the target memory unit.

[0012] In some embodiments, each level of the multi-level bypass unit includes: a bypass marking module configured to: store marking information of the bypass unit at this level, the marking information indicating address information and validity status information, the address information being associated with updated data stored in the bypass unit at this level, and the validity status information indicating the validity status of the updated data stored in the bypass data storage module; a bypass data storage module configured to: store the updated data stored in the bypass unit at this level; and a comparator configured to: compare the currently requested address information with the address information in the bypass marking module.

[0013] In some embodiments, the multi-level bypass unit is further configured to: while backing up the update data associated with the previous request to the bypass unit of a predetermined level in the multi-level bypass unit, back up the update data already backed up in the bypass unit of the corresponding level in the multi-level bypass unit to the bypass unit of the next level below the corresponding level.

[0014] In some embodiments, the multi-level bypass unit is further configured to: in response to determining that there is no data dependency between the current request to be executed and the previous request, obtain updated data of the current request from the target storage unit for use in executing the current request.

[0015] In some embodiments, each of the multi-level bypass units stores at least tagging information and update data. The tagging information includes: validity status information indicating the update data stored by the bypass unit at that level, and address information about the update data.

[0016] In some embodiments, the predetermined level of bypass unit is the first level of bypass unit in a multi-level bypass unit.

[0017] In some embodiments, the method for executing a host-issued request further includes: while backing up the update data associated with the previous request to a predetermined level of the multi-level bypass unit, backing up the update data already backed up in the corresponding level of the multi-level bypass unit to the next level of the bypass unit.

[0018] In some embodiments, configuring a bypass unit in an on-chip region adjacent to the computing core includes: configuring multiple levels of bypass units in an on-chip region adjacent to the computing core based on the difference between the number of clock cycles for reading data and updating data for the target memory unit by the computing core, wherein the number of levels of the bypass units is equal to the difference.

[0019] In some embodiments, determining whether there is a data dependency between the current request and a previous request includes: determining, via a comparator included in the bypass unit of the corresponding level, whether the address information of the current request is the same as the address information in the bypass unit of the level corresponding to the previous request, the address information being stored by a bypass tag module in the bypass unit of the level corresponding to the previous request.

[0020] In some embodiments, in response to determining that the current request to be executed has a data dependency with a previous request, obtaining the backed-up updated data of the previous request from the bypass unit of the corresponding level of the previous request with the data dependency includes: in response to determining that the current request has a data dependency with the previous request, obtaining the backed-up updated data of the previous request from the bypass unit of the first level.

[0021] In some embodiments, in response to determining that the current request to be executed has a data dependency with a previous request, obtaining the backed-up updated data of the previous request from the bypass unit at the corresponding level of the previous request with the data dependency includes: in response to determining that the current request does not have a data dependency with the previous request, determining that the current request has a data dependency with the request two years prior, wherein the request two years prior is the adjacent request preceding the previous request; and in response to determining that the current request has a data dependency with the request two years prior, obtaining the backed-up updated data of the request two years prior from the bypass unit at the second level.

[0022] In some embodiments, the method for executing a host-issued request further includes: in response to determining that the current request to be executed has no data dependency with a previous request, obtaining updated data of the current request from a target storage unit for executing the current request.

[0023] This invention establishes multiple levels of bypass units in the on-chip region adjacent to the computing core of an artificial intelligence chip. These bypass units back up the updated data associated with the previous request to a predetermined level of bypass unit, thus ensuring timely backup of request update data. Furthermore, the multiple levels of bypass units are configured to, in response to determining a data dependency between the current request and a previous request, retrieve the backed-up updated data of the previous request from the corresponding level of the bypass unit of the dependent previous request for use in executing the current request. Therefore, even when a data dependency exists between the current and previous requests, the data of the previous request can be directly read from the corresponding bypass unit, ensuring uninterrupted request processing pipeline operation. Thus, this invention enables uninterrupted request processing pipeline operation even when a data dependency exists between the current and previous requests, significantly improving chip processing efficiency. In addition, the present invention configures multi-level bypass units in the on-chip region adjacent to the computing core, which can significantly shorten the backup and read path of requested update data, thus helping to reduce power consumption and memory access latency, and further improve the processing efficiency of the chip.

[0024] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0025] The above and other features, advantages, and aspects of the various embodiments of the present invention will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements.

[0026] Figure 1 The illustration shows a schematic diagram of a computing device for implementing a method for executing a host-issued request according to an embodiment of the present invention.

[0027] Figure 2 A flowchart of a method for executing a host-issued request according to an embodiment of the present invention is shown.

[0028] Figure 3 A schematic diagram of an artificial intelligence chip according to some embodiments of the present invention is shown.

[0029] Figure 4 A schematic diagram of the structure of a multi-stage bypass unit according to some embodiments of the present invention is shown.

[0030] Figure 5 A flowchart of a method for determining whether there is a data dependency between a current request to be executed and a previous request, according to an embodiment of the present invention, is shown.

[0031] Figure 6 A flowchart illustrating data backup and data dependency determination according to the present invention is shown.

[0032] In the various figures, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0033] Preferred embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While preferred embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that the invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0034] The term "comprising" and its variations as used herein signify open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "at least partially based on". The terms "one example embodiment" and "one embodiment" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc., may refer to different or the same objects.

[0035] As described above, the shortcomings of traditional artificial intelligence chips and methods for executing host-issued requests are that when there are data dependencies between requests, the request processing pipeline is interrupted, affecting the chip's processing efficiency.

[0036] To at least partially address one or more of the aforementioned problems and other potential issues, exemplary embodiments of the present invention provide an artificial intelligence chip. In this AI chip, the chip includes: a computing core configured to execute requests issued by a host; a target storage unit configured to store update data associated with the request; and a multi-level bypass unit disposed in an on-chip region adjacent to the computing core, the multi-level bypass unit being configured to: back up update data associated with a previous request to a predetermined level of the multi-level bypass unit; thus, the requested update data can be backed up to the bypass unit in a timely manner. Furthermore, the multi-level bypass units are configured to: in response to determining that the current request to be executed has a data dependency with a previous request, retrieve the backed-up updated data of the previous request from the corresponding level of the bypass unit of the previous request with the data dependency, and use this data to execute the current request; therefore, when there is a data dependency between the current request and the previous request, the data of the previous request can be read directly from the corresponding bypass unit, ensuring that the request processing pipeline is not interrupted. Thus, even when there is a data dependency between the current request and the previous request, the present invention can ensure that the request processing pipeline operates without interruption, significantly improving the chip's processing efficiency. In addition, by configuring the multi-level bypass units in the on-chip region adjacent to the computing core, the present invention can significantly shorten the backup and read path of the updated data of the request, thereby helping to reduce power consumption and memory access latency, and further improving the chip's processing efficiency.

[0037] Figure 1 A schematic diagram illustrates a computing device 100 implementing a method for executing a host-issued request according to an embodiment of the present invention. Figure 1As shown, the computing device 100 may have one or more processing units and a memory communicatively connected to one or more processors; wherein the memory stores instructions executable by at least one processor. The processing units include, for example, dedicated processing units such as Graphics Processing Units (GPUs), Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), General-purpose computing on graphics processing units (GPGPUs), and general-purpose processing units such as CPUs. The computing device 100 also includes at least: a requested update data backup unit 102, a data dependency determination unit 104, and a backed-up update data reading unit 106.

[0038] Regarding the update data backup unit 102 for the request, it is used to store the update data associated with the previous request issued by the host to the target storage unit, and at the same time, back up the update data associated with the previous request to a predetermined level of the multi-level bypass unit, wherein the multi-level bypass unit is configured in the on-chip region adjacent to the computing core.

[0039] Regarding the data dependency determination unit 104, it is used to determine whether there is a data dependency between the current request to be executed and the previous request.

[0040] Regarding the backed-up update data reading unit 106, in response to determining that the current request to be executed has a data dependency with the previous request, it retrieves the backed-up update data of the previous request from the bypass unit at the corresponding level of the previous request with the data dependency, so as to execute the current request.

[0041] The following will combine Figure 2 , Figure 3 and Figure 4 This invention describes a method 200 for executing a host-issued request, according to embodiments of the present invention. It should be understood that method 200 can, for example, be implemented in... Figure 1 The described computing device 100 is executed. Method 200 can also be performed, for example, at... Figure 3 The described artificial intelligence chip 300 is executed. Method 200 may also include additional actions not shown and / or the actions shown may be omitted, and the scope of the invention is not limited in this respect.

[0042] It should be understood that Figure 3The AI ​​chip shown is merely a specific implementation of the computing device 100; the computing device 100 is not limited to... Figure 3 The specific structure shown. The computing device 100 may also include... Figure 3 Other components besides the AI ​​chip shown.

[0043] In step 202, while storing the update data associated with the previous request issued by the host to the target storage unit, the computing device 100 also backs up the update data associated with the previous request to a predetermined level of bypass unit in a multi-level bypass unit, wherein the multi-level bypass unit is configured in an on-chip region adjacent to the computing core.

[0044] like Figure 3 As shown, the artificial intelligence chip 300 includes shared memory 302, a secondary cache 304, and multiple computing cores (e.g., Figure 3 The example shows computing core 321, as well as other computing cores. Each computing core is associated with a controller, a target memory unit, and multiple levels of bypass units. For example, computing core 321 is associated with controller 311, target memory unit 331, and multiple levels of bypass units 400.

[0045] Regarding the target storage unit, it may be, for example, a level 1 cache. In some embodiments, the target storage unit is Static Random Access Memory (SRAM). It should be understood that the target storage unit is configured to store the updated data associated with the request.

[0046] The computing core is configured to execute requests issued by the host.

[0047] Regarding bypass units, they are configured in an on-chip region adjacent to the computing core. For example, based on the difference between the number of clock cycles for reading data and updating data for a target memory cell by the computing core (e.g., the difference is M, where M is a natural number), multiple levels of bypass units are configured in an on-chip region adjacent to the computing core, the number of levels of the bypass units being equal to the difference M.

[0048] For example, please refer to Figure 4 , Figure 4 The diagram illustrates the specific structure and related data flow of the multi-level bypass unit 400. The multi-level bypass unit 400 contains M levels of bypass units (e.g., bypass unit 0, bypass unit 1, ..., bypass unit M-1). For example, if the difference M = 2 between the clock cycles for reading and updating data from the target memory unit 331 by the computing core 321, then the multi-level bypass unit 400 is configured as two levels, with the first level being, for example, bypass unit 0, and the second level being, for example, bypass unit 1.

[0049] Regarding the predefined level of bypass unit, for example, the first level of bypass unit in a multi-level bypass unit. For example, the update data associated with the previous request is backed up to the first level of bypass unit (e.g., bypass unit 0) in the multi-level bypass unit 400.

[0050] In some embodiments, while backing up the update data associated with the previous request to a predetermined level of the multi-level bypass unit, the update data already backed up in the corresponding level of the multi-level bypass unit is backed up again to the next lower level of the bypass unit. For example, continuing the above example, the update data associated with the previous request is backed up to the first level of the multi-level bypass unit 400 (e.g., bypass unit 0), and at the same time, the update data already backed up in the first level of the bypass unit (e.g., bypass unit 0) (which is, for example, the update data associated with the adjacent request before the previous request) is backed up again to the second level of the bypass unit (e.g., bypass unit 1).

[0051] In some embodiments, each bypass unit in a multi-level bypass unit stores at least tagging information and update data. The tagging information includes: validity status information indicating the update data stored by the bypass unit at that level, and address information about the update data. For example, please refer to... Figure 4 The first-level bypass unit (e.g., bypass unit 0) includes a bypass unit 0 comparator, a bypass unit 0 bypass flag module, and a bypass unit 0 bypass data storage module. Update data is stored in the bypass unit 0 bypass data storage module, while address information and validity status information are stored in the bypass unit 0 bypass flag module.

[0052] At step 204, the computing device 100 determines whether there is a data dependency between the current request to be executed and the previous request.

[0053] A prior request is, for example, a request that precedes the current request. It should be understood that a prior request includes at least the previous request.

[0054] A method for determining whether there is a data dependency between a current request to be executed and a previous request includes, for example, a computing device 100 determining, via a comparator included in a bypass unit of a corresponding level, whether the address information of the current request is the same as the address information in the bypass unit of the level corresponding to the previous request, the address information being stored by a bypass tag module in the bypass unit of the level corresponding to the previous request.

[0055] For example, if the computing device 100 determines that the address information of the current request is the same as the address information in the bypass unit corresponding to the level of the previous request, it determines that the current request to be executed has a data dependency with the previous request; if the computing device 100 determines that the address information of the current request is different from the address information in the bypass unit corresponding to the level of the previous request, it determines that the current request to be executed does not have a data dependency with the previous request. The following will combine... Figure 5 The specific details of method 500, used to determine whether there is a data dependency between the current request to be executed and the previous request, will not be elaborated here.

[0056] At step 206, if the computing device 100 determines that the current request to be executed has a data dependency with a previous request, it obtains the backed-up updated data of the previous request from the bypass unit of the corresponding level of the previous request with the data dependency, so as to execute the current request.

[0057] For example, please refer to Figure 4 The first-level bypass unit (e.g., bypass unit 0) determines, via the bypass unit 0 comparator, whether the address information of the current request is the same as the address information stored in the bypass tag module of bypass unit 0 (the bypass tag module of bypass unit 0 stores the address information of the updated data of the previous request). If the bypass unit 0 comparator determines that the address information is the same, it determines that there is a data dependency between the current request and the previous request, and obtains the updated data of the previous request from the bypass data storage module of bypass unit 0 (i.e., the bypass data storage module included in the first-level bypass unit) to execute the current request.

[0058] In the above method, this invention sets up multi-level bypass units, enabling timely backup of updated data associated with previous requests that have not yet been updated in the target storage unit due to latency. This allows the AI ​​chip to directly read updated data about previous requests from the multi-level bypass units when there is a data dependency between the current request and previous requests, without waiting for the relevant data in the target storage unit to be updated. This ensures uninterrupted operation of the request processing pipeline, allowing the computing units within the computing core to quickly obtain the required data without being affected by data dependencies, significantly improving the chip's processing efficiency. Furthermore, by configuring the multi-level bypass units in an on-chip region adjacent to the computing core, this invention significantly shortens the backup and read path for requested updated data, thus reducing power consumption and memory access latency, further improving the chip's processing efficiency.

[0059] The following will combine Figure 3 , Figures 4 to 5 This invention describes an embodiment of a method 500 for determining whether a current request to be executed has a data dependency with a previous request. It should be understood that method 500 can, for example, be implemented in... Figure 1The described computing device 100 is executed. Method 500 can also be implemented, for example, at... Figure 3 The described artificial intelligence chip 300 is executed. Method 500 may also include additional actions not shown and / or the actions shown may be omitted; the scope of the invention is not limited in this respect.

[0060] At step 502, if the computing device 100 determines that there is a data dependency between the current request and the previous request, it retrieves the backed-up updated data of the previous request from the first-level bypass unit.

[0061] For example, calculate the difference M between the number of clock cycles required for the kernel to read data from and update data in the target memory cell. Please refer to... Figure 4 The multi-level bypass units are: the first-level bypass unit (e.g., bypass unit 0), the second-level bypass unit (e.g., bypass unit 1), and so on up to the (M-1)th-level bypass unit (e.g., bypass unit M-1). For example, the current request is a request with N+M clock cycles (where N and M are natural numbers). The previous request of the current request is, for example, a request with N+M-1 clock cycles. For example, if it is determined that there is a data dependency between the request with N+M clock cycles and the request with N+M-1 clock cycles, the backed-up updated data of the previous request is obtained from the first-level bypass unit (e.g., bypass unit 0). It should be understood that the methods for determining whether there is a data dependency between the request with N+M clock cycles and the request with N+M-1 clock cycles include, for example, methods such as... Figure 4 As indicated by 402, the comparator of the first-stage bypass unit (e.g., the bypass unit 0 comparator) determines whether the requested address information for the number of clock cycles N+M is the same as the address information in the bypass tag module of the first-stage bypass unit (e.g., the bypass unit 0 bypass tag module).

[0062] At step 504, if the computing device 100 determines that there is no data dependency between the current request and the previous request, it determines whether there is a data dependency between the current request and the request before that, where the request before that is the adjacent request preceding the previous request.

[0063] For example, continuing the example above, if it is determined that there is no data dependency between the request with clock cycle number N+M and the request with clock cycle number N+M-1, then it is determined whether there is a data dependency between the request with clock cycle number N+M and the request with clock cycle number N+M-2. For example, as... Figure 4 As indicated by 404, the comparator (e.g., the comparator of the second-level bypass unit (e.g., the comparator of the bypass unit 1) is used to determine whether the address information requested for the number of clock cycles N+M is the same as the address information in the bypass tag module (e.g., the bypass tag module of the bypass unit 1) of the second-level bypass unit.

[0064] At step 506, if the computing device 100 determines that there is a data dependency between the current request and the request two years prior, it obtains the backed-up updated data of the request two years prior from the bypass unit of the second level.

[0065] For example, continuing the example above, if a data dependency exists between a request for clock cycles N+M and a request for clock cycles N+M-2, please refer to... Figure 4 The backup updated data of the previous request (e.g., clock cycle number N+M-2) is obtained from the second-level bypass unit (e.g., bypass unit 1).

[0066] At step 508, if the computing device 100 determines that the current request to be executed has no data dependency with the previous request, it obtains the updated data of the current request from the target storage unit for use in executing the current request.

[0067] For example, continuing the above example, if it is determined that there is no data dependency between the request for clock cycles N+M and the request for clock cycles N+M-2, then a data dependency determination is performed between the request for clock cycles N+M and the request for clock cycles N+M-3. For example, it is determined whether there is a data dependency between the request for clock cycles N+M and the request for clock cycles N+M-3. For example, via the comparator included in the third-level bypass unit (e.g., bypass unit 2 comparator), it is determined whether the address information of the request for clock cycles N+M is the same as the address information in the bypass flag module of the third-level bypass unit (e.g., bypass unit 2 bypass flag module). If there is a data dependency between the request for clock cycles N+M and the request for clock cycles N+M-3, the backed-up updated data regarding the request for clock cycles N+M-3 is obtained from the third-level bypass unit (e.g., bypass unit 2). If there is no data dependency between the current request and the previous requests of the previous M clock cycles, for example, if... Figure 4 As indicated by 406, the address information of the request for clock cycle number N+M, determined by the comparator (e.g., the comparator of the bypass unit M-1) included in the bypass unit of the Mth stage (e.g., the bypass unit M-1), is also different from the address information in the bypass marker module of the bypass unit of the Mth stage (e.g., the bypass marker module of the bypass unit M-1). Updated data of the current request is obtained from the target memory unit (e.g., the target memory unit 331) for the purpose of executing the current request.

[0068] In the above solution, the present invention can obtain the required updated data from the bypass unit or target storage unit corresponding to the previous request that has a data dependency relationship, depending on whether there is a data dependency relationship between the current request and the previous request. This allows the request processing pipeline to work uninterruptedly and is not affected by data dependency, thus significantly improving the processing efficiency of the chip.

[0069] The following combination Figure 3 , Figure 4 and Figure 6 The structure and function of the artificial intelligence chip 300 and the multi-level bypass unit 400 are described in detail.

[0070] like Figure 3 As shown, the artificial intelligence chip 300 includes at least shared memory 302, a secondary cache 304, and a computing core 321. Figure 3 The example illustrates multiple compute cores, a target storage unit 331, and multiple levels of bypass units 400. Compute core 321 is configured, for example, to execute requests issued by the host. The target storage unit 331 (e.g., a level 1 cache) is configured, for example, to store the update data associated with the request. The multiple levels of bypass units 400 are located, for example, in an on-chip region adjacent to compute core 321, and are constructed, for example, by buffers. The multiple levels of bypass units are configured to: back up the update data associated with the previous request to a predetermined level of bypass units within the multiple levels of bypass units; and, in response to determining that the current request to be executed has a data dependency with a previous request, retrieve the backed-up update data of the previous request from the corresponding level of the bypass unit of the previous request with the data dependency, for use in executing the current request.

[0071] In some embodiments, each stage of the multi-stage bypass unit 400 includes a bypass tag module, a bypass data storage module, and a comparator. For example, please refer to... Figure 4 The multi-level bypass unit 400 includes M levels of bypass units (e.g., bypass unit 0, bypass unit 1, ..., bypass unit M-1). For example, the first-level bypass unit (e.g., bypass unit 0) includes: a bypass unit 0 comparator, a bypass unit 0 bypass flag module, and a bypass unit 0 bypass data storage module.

[0072] Regarding the bypass marking module, it is configured, for example, to store marking information of the bypass units at this level, wherein the marking information indicates address information and validity status information, the address information is associated with the updated data stored by the bypass unit at this level, and the validity status information indicates the validity status of the updated data stored by the bypass data storage module. For example, please refer to... Figure 4The bypass unit 0 bypass tagging module is configured to store tagging information of the first-level bypass unit (e.g., bypass unit 0). For example, the tagging information of the first-level bypass unit (e.g., bypass unit 0) is the address information and validity status information of the update data associated with the previous request.

[0073] Regarding the bypass data storage module, it is configured, for example, to store updated data stored by the bypass unit at this level. For example, please refer to... Figure 4 The bypass unit 0 bypass data storage module is configured to store the updated data stored in the first-level bypass unit (e.g., bypass unit 0).

[0074] Regarding the comparator, it is configured, for example, to compare the address information of the currently requested address with the address information in the bypass flag module. For example, please refer to... Figure 4 The bypass unit 0 comparator is configured to compare the address information of the current request with the address information in the bypass unit 0 bypass flag module. It should be understood that if the address information of the current request is the same as the address information in the bypass unit 0 bypass flag module, it indicates that the current request has a data dependency on the previous request.

[0075] In some embodiments, the multi-level bypass unit 400 is further configured to: while backing up the update data associated with the previous request to a predetermined level of the multi-level bypass unit 400, simultaneously back up the update data already backed up in the corresponding level of the multi-level bypass unit 400 to the next lower level of the bypass unit. For example, please refer to... Figure 4 The pre-defined level is the first level. When backing up the update data associated with the previous request to the first level bypass unit (e.g., bypass unit 0), the update data of the adjacent request before the previous request (i.e., the request before the previous one) that was originally backed up in the first level bypass unit is backed up again to the second level bypass unit (e.g., bypass unit 1).

[0076] In some embodiments, the AI ​​chip 300 further includes a data selector 410, configured to determine corresponding data (as indicated by label 408) from the data storage modules of the corresponding level of the bypass units in the target storage unit 331 and the multi-level bypass units 400 based on a data dependency judgment result between the current request and a previous request, for use by the computing core 321 to execute the current request. It should be understood that in some embodiments, a data selector may not be provided.

[0077] Please refer to Figure 6 This further illustrates the multi-level bypass unit 400. Taking the difference M between the number of clock cycles for reading and updating data from the target memory unit 331 by the calculation core 321 as an example.

[0078] The current clock cycle number is N+M. The update data of the previous request (i.e., the request with clock cycle number N+M-1) is backed up to the first-level bypass unit (e.g., bypass unit 0) (as indicated by label 612). Bypass unit 0 recursively pushes the backup data of its original stored update data to the next-level bypass unit (e.g., bypass unit 1) (as indicated by label 614). Each level of bypass unit backs up the backup data of its original stored update data to the next-level bypass unit, until the backup reaches the M-th level bypass unit (e.g., bypass unit M-1). The request update data for the current clock cycle is stored in the target storage unit 331 (as indicated by label 622).

[0079] For the current request and the previous request, a data dependency determination is performed. The first-level bypass unit (e.g., bypass unit 0) uses a comparator in bypass unit 0 to determine if a data dependency exists between the current request and the previous request. If a data dependency exists, the current request is considered to have a data dependency scenario (fwd01), and the corresponding data is retrieved from the bypass data storage module of bypass unit 0, i.e., updated data about the previous request is retrieved. If no data dependency exists between the current request and the previous request, the second-level bypass unit (e.g., bypass unit 1) uses a comparator in bypass unit 1 to determine if there is a data dependency between the current request and the request two years prior. If a data dependency exists between the current request and the request two years prior, the dependency is determined. If a data dependency is found between the current request and the previous request (fwd02), the corresponding data is retrieved from the bypass data storage module of bypass unit 1, i.e., updated data about the previous request. If no data dependency exists between the current request and the previous request, the process continues through the next-level bypass unit comparator to determine if a data dependency exists between the current request and the request before the previous request, and so on, until the data dependency is determined by the bypass unit M-1 comparator of the M-th level bypass unit. If a data dependency exists between the current request and the previous M requests, a data dependency scenario (fwd0M) is identified, and the corresponding data is retrieved from the bypass data storage module of bypass unit M-1, i.e., updated data about the previous M requests. If a data dependency exists between the current request and any of the previous requests up to the previous M requests, the corresponding data is retrieved from the bypass data storage module of the bypass unit corresponding to the previous request with which the data dependency exists, to be used to execute the current request. The fwd01 data dependency scenario indicates a data dependency between the current request and the previous request, requiring the updated data of the previous request to be obtained from the first-level bypass unit (e.g., bypass unit 0). The fwd02 data dependency scenario indicates a data dependency between the current request and the request two levels prior, requiring the updated data of the request two levels prior to that point to be obtained from the second-level bypass unit (e.g., bypass unit 1). The fwd03 data dependency scenario indicates a data dependency between the current request and the request before the request two levels prior to that point, requiring the updated data of the request two levels prior to that point to be obtained from the third-level bypass unit (e.g., bypass unit 2). The fwd0M data dependency scenario indicates a data dependency between the current request and the Mth preceding request (or "the Mth previous request"), requiring the updated data of the Mth preceding request to be obtained from the Mth-level bypass unit (e.g., bypass unit M-1).

[0080] In some embodiments, the multi-level bypass unit 400 is further configured to: in response to determining that there is no data dependency between the current request to be executed and the previous request, obtain updated data of the current request from the target storage unit 331 for use in executing the current request. For example, if there is no data dependency between the current request and the previous request, then obtain the corresponding data (as indicated by label 624) from the target storage unit 331 for use in executing the current request.

[0081] Therefore, by setting up multi-level bypass units, this invention can classify the data dependencies between the current request and previous requests into corresponding types (e.g., fwd01 data dependency scenario, fwd02 data dependency scenario, ..., fwd0M data dependency scenario). If a data dependency scenario exists, the corresponding data is obtained from the bypass unit at the corresponding level; if no data dependency scenario exists, the corresponding data is obtained from the target storage unit. Thus, regardless of how complex the data dependency relationship between the current request and previous requests is, it is only necessary to determine the data dependency relationship between the current request and previous requests level by level, and directly obtain the updated data of the dependent previous requests from the bypass unit at the corresponding level. This allows the request processing pipeline to work uninterrupted, enabling the computing units within the computing core to quickly obtain the required data without being affected by data dependencies, significantly improving the chip's processing efficiency.

[0082] The various processes and procedures described above, such as methods 200 and 500, can be executed at a computing device. This computing device may include, for example, at least one processor (at least one graphics processor and at least one central processing unit); and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor. In some embodiments, methods 200 and 500 may be implemented as a computer software program or program product tangibly contained in a machine-readable medium. In some embodiments, part or all of the computer program may be loaded and / or installed on the computing device via read-only memory (ROM) and / or a communication unit. When the computer program is loaded into random-access memory (RAM) and executed by the GPU and CPU, one or more actions of methods 200 and 500 described above can be performed.

[0083] This invention can be a method, apparatus, system, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions used by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof.

[0084] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network, to an external computer or external storage device. Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0085] These computer-readable program instructions can be provided to the central processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the central processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0086] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0087] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps loaded in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0088] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors.

Claims

1. An artificial intelligence chip, characterized in that, include: The computing core is configured to execute requests issued by the host. The target storage unit is configured to store the updated data associated with the request; Multi-level bypass units are located in on-chip regions adjacent to computing cores, and the multi-level bypass units are configured to back up the update data associated with the previous request to a predetermined level of bypass unit in the multi-level bypass units. In response to determining that the current request to be executed has a data dependency with a previous request, the backed-up updated data of the previous request is obtained from the bypass unit at the corresponding level of the previous request with the data dependency, so as to be used to execute the current request.

2. The artificial intelligence chip according to claim 1, characterized in that, The number of levels of the multi-level bypass unit is equal to the difference between the number of clock cycles the computing core uses to read and update data for the target storage unit.

3. The artificial intelligence chip according to claim 1, characterized in that, Each of the multi-level bypass units includes: The bypass marking module is configured to: store marking information of the bypass unit at this level, wherein the marking information indicates address information and validity status information, wherein the address information is associated with the updated data stored by the bypass unit at this level, and the validity status information indicates the validity status of the updated data stored by the bypass data storage module; The bypass data storage module is configured to store the updated data stored by the bypass unit at this level; The comparator is configured to compare the address information of the currently requested address with the address information in the bypass marker module.

4. The artificial intelligence chip according to claim 1, characterized in that, The multi-level bypass unit is further configured to: while backing up the update data associated with the previous request to the bypass unit of a predetermined level in the multi-level bypass unit, back up the update data already backed up in the corresponding level's bypass unit to the next level's bypass unit.

5. The artificial intelligence chip according to claim 1, characterized in that, The multi-level bypass unit is further configured to: in response to determining that the current request to be executed has no data dependency with the previous request, obtain the updated data of the current request from the target storage unit for use in executing the current request.

6. A method for executing a host-issued request, characterized in that, include: While storing the update data associated with the previous request issued by the host to the target storage unit, the update data associated with the previous request is backed up to a predetermined level of bypass unit in a multi-level bypass unit, wherein the multi-level bypass unit is configured in an on-chip region adjacent to the computing core. Determine if there is a data dependency between the current request to be executed and previous requests; as well as In response to the determination that the current request to be executed has a data dependency with a previous request, the backed-up updated data of the previous request is obtained from the bypass unit at the corresponding level of the previous request with the data dependency, so as to be used to execute the current request.

7. The method according to claim 6, characterized in that, Each of the multi-level bypass units stores at least tagging information and update data. The tagging information includes: validity status information indicating the update data stored in the current level of the bypass unit, and address information about the update data.

8. The method according to claim 6, characterized in that, The pre-defined bypass unit is the first-level bypass unit in a multi-level bypass unit system.

9. The method according to claim 6, characterized in that, Also includes: While backing up the update data associated with the previous request to the predetermined level of the multi-level bypass unit, the update data already backed up in the corresponding level of the multi-level bypass unit is backed up again to the next level of the bypass unit.

10. The method according to claim 6, characterized in that, The bypass unit is configured in an on-chip region adjacent to the computing core, including: Based on the difference between the number of clock cycles for reading data and updating data in the target memory cell by the computing core, a multi-level bypass unit is configured in the on-chip region adjacent to the computing core, and the number of levels of the multi-level bypass unit is equal to the difference.

11. The method according to claim 6, characterized in that, Determining whether there is a data dependency between the current request and previous requests includes: The comparator included in the corresponding level of the bypass unit determines whether the address information of the current request is the same as the address information in the bypass unit of the previous request level, which is stored by the bypass flag module in the bypass unit of the previous request level.

12. The method according to claim 6, characterized in that, In response to determining that the current request to be executed has a data dependency with a previous request, retrieving the backed-up updated data of the previous request from the corresponding level of the bypass unit of the previous request with the data dependency includes: In response to the determination that there is a data dependency between the current request and the previous request, the backed-up updated data of the previous request is obtained from the first-level bypass unit.

13. The method according to claim 6, characterized in that, In response to determining that the current request to be executed has a data dependency with a previous request, retrieving the backed-up updated data of the previous request from the corresponding level of the bypass unit of the previous request with the data dependency includes: In response to determining that there is no data dependency between the current request and the previous request, it is determined whether there is a data dependency between the current request and the request two years prior, where the request two years prior is the adjacent request preceding the previous request; and In response to the determination that there is a data dependency between the current request and the request two hours ago, the updated data of the previous request that has been backed up is obtained from the bypass unit of the second level.

14. The method according to claim 6, characterized in that, Also includes: In response to the determination that there is no data dependency between the current request to be executed and the previous request, the updated data of the current request is obtained from the target storage unit to be used to execute the current request.

15. A computing device, characterized in that, include: At least one actuator; as well as A memory communicatively connected to the at least one actuator; in The memory stores instructions that can be executed by the at least one actuator to enable the at least one actuator to perform the method of any one of claims 6-14.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a machine, performs the method according to any one of claims 6-14.

17. A computer program product, characterized in that, Includes a computer program, which, when executed by a machine, performs the method according to any one of claims 6-14.

Images