A method for caching request fusion and an artificial intelligence chip

Abstract

This application provides a cache request fusion method and an artificial intelligence chip, relating to the field of artificial intelligence chip technology. The method includes: querying an incomplete buffer based on a first memory address carried by a first data request; when at least one second data request is found at the same memory address, i.e., when the first data request and at least one second data request need to operate on the same cache line, merging the first data request and at least one second data request into a single target data request; executing the target data request obtains the response content of both the first data request and at least one second data request, thus reducing the number of data requests executed; correspondingly, the number of resource consumption operations necessary for executing data requests is also reduced, thereby reducing resource waste and improving internal bandwidth. Merging multiple data requests into a single data request increases the amount of memory accessed in a single request, thereby improving cache performance and memory access efficiency.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence chip technology, and in particular to a cache request fusion method and an artificial intelligence chip. Background Technology

[0002] In artificial intelligence chips, the memory access path for the Last Level Cache (LLC) generally uses a large size, meaning that the amount of data in a single cache line in the LLC is large. At the same time, in order to reduce implementation complexity, there is generally only one hit-miss check pipeline. Therefore, increasing the amount of memory data accessed by a single request is the key to improving the speed of the Last Level Cache.

[0003] Under related technologies, artificial intelligence chips mostly adopt a Single Instruction, Multiple Threads (SIMT) architecture. Under this architecture, there may be multiple requests of small size (i.e., the amount of memory accessed in the request is much smaller than the amount of data in a single cache line in LLC), which leads to resource waste and consequently reduces cache performance. Summary of the Invention

[0004] This application provides a cache request fusion method and an artificial intelligence chip, which are used to merge multiple data requests with the same address into a single data request for memory access, thereby reducing resource consumption and improving cache performance.

[0005] On one hand, embodiments of this application provide a cached request fusion method, the method comprising:

[0006] Receive a first data request from the first processing core, wherein the first data request includes a first memory address;

[0007] When there is at least one second data request corresponding to the first memory address in the incomplete buffer, the first data request and the at least one second data request are merged into a target data request. The incomplete buffer is used to store cached data requests that have been received but not completed.

[0008] Execute the target data request to obtain a target request response; the target request response includes the response content of the first data request and the at least one second data request.

[0009] Optionally, when there is at least one second data request corresponding to the first memory address in the uncompleted buffer, merging the first data request and the at least one second data request into a single target data request includes:

[0010] When there is at least one second data request corresponding to the first memory address in the incomplete buffer, the second data request that has been issued and executed among the at least one second data request shall be taken as the target data request;

[0011] Set the first data request to a waiting state.

[0012] Optionally, the first data request further includes: a first request identifier; after receiving the first data request from the first processing core, it further includes:

[0013] A first information record for the first data request is added to the incomplete buffer. The first information record includes: the first request identifier and the first memory address.

[0014] Optionally, after executing the target data request and obtaining the target request response, the method further includes:

[0015] The response content of the first data request is returned to the first processing core;

[0016] The response content for each second data request is returned to the corresponding second processing core.

[0017] Optionally, the incomplete buffer is a list of incomplete reads, and the first data request and the second data request are read requests;

[0018] The execution of the target data request and the acquisition of the target request response include:

[0019] When the target data request hits a cache line, the data of the cache line is read from the cache memory based on the tag information of the cache line;

[0020] The data of the cached line is added to the first buffer in the read data cache as the response to the target request.

[0021] Optionally, the list of reads not completed includes: the target information record of the target data request;

[0022] After adding the data of the cached line as the target request response to the first buffer in the read data cache, the method further includes:

[0023] Add a target pointer to the first buffer to the target information record;

[0024] The response content of the target data request is read from the first buffer according to the target pointer and the memory access offset carried by the target data request.

[0025] Optionally, the list of reads not completed may further include: the first information record of the first data request;

[0026] After adding the data of the cached line as the target request response to the first buffer in the read data cache, the method further includes:

[0027] Add a target pointer to the first buffer to the first information record;

[0028] The response content of the first data request is read from the first buffer according to the target pointer and the memory access offset carried by the first data request.

[0029] Optionally, the incomplete buffer is a list of incomplete writes; the first data request and the second data request are write requests; the first data request further includes: an input buffer address and a first set of data to be written;

[0030] After receiving the first data request from the first processing core, the process further includes:

[0031] Write the first data to be written to the second buffer corresponding to the input buffer address.

[0032] Optionally, executing the target data request and obtaining the target request response includes:

[0033] When the target data request hits a cache line, read the full data already stored in the second buffer; and write the full data into the storage area corresponding to the cache line in the cache memory;

[0034] After a successful write operation, a write success message will be sent as the response to the target request.

[0035] Optionally, after successfully writing the data and sending a write success message as a response to the target request, the method further includes:

[0036] The write success message is used as the response content for the first data request and the at least one second data request, respectively.

[0037] On one hand, embodiments of this application provide an artificial intelligence chip, including:

[0038] Processes the core, cache, and main memory;

[0039] The cache is used to execute the aforementioned cache request fusion method.

[0040] On one hand, embodiments of this application provide a computer device, including a memory, an artificial intelligence chip, and a computer program stored on the memory and executable on the artificial intelligence chip. When the artificial intelligence chip executes the computer program, it implements the steps of the above-mentioned cache request fusion method.

[0041] In this embodiment, the cache receives a first data request sent by the first processing core, and then queries the incomplete buffer based on the first memory address carried by the first data request. The incomplete buffer is used to store data requests received by the cache that are not yet completed. When at least one second data request with the same memory address is found, that is, when the first data request and at least one second data request need to operate on the same cache line, the first data request and at least one second data request are merged into a single target data request. Then, the target data request is executed to obtain the response content of the first data request and at least one second data request, thus reducing the number of data requests executed. Correspondingly, the number of times the resources necessary to execute data requests (such as hit check resources, bandwidth resources, and other resources) are consumed is also reduced, thereby reducing resource waste and improving internal bandwidth.

[0042] Secondly, merging multiple data requests into a single request increases the amount of data accessed in memory for a single request, thereby improving cache performance and memory access efficiency. Furthermore, the cache request fusion method in this application only requires adding relevant logic when the data request enters the cache; the implementation logic is simple and does not increase the design complexity of the cache, making it easy to implement. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;

[0045] Figure 2 This is a schematic diagram of the structure of an artificial intelligence chip provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of the structure of a merging engine provided in an embodiment of this application;

[0047] Figure 4 A flowchart illustrating a cached request fusion method provided in an embodiment of this application;

[0048] Figure 5This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0049] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0050] The terms "first," "second," "third," etc., used in this application are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Furthermore, in the description of embodiments in this application, unless otherwise stated, "a plurality of" means two or more.

[0051] Under the relevant technologies and the SIMT architecture, during the process of the core accessing the LLC, there may be multiple requests of smaller size (i.e., the amount of memory accessed in the request is much smaller than the amount of data in a single cache line in the LLC). Since the cache line is the smallest unit of operation in the cache, even if the amount of memory accessed in the current request is much smaller than the amount of data in a single cache line in the LLC, the cache will still process it at the cache line granularity.

[0052] For example, if the cache line contains 512 bytes of data, and the read request requires 128 bytes of memory access data, the cache will read 512 bytes of data and return them to the processing core. The processing core will then select the necessary 128 bytes from the returned data, and the rest of the returned data will not actually be needed.

[0053] This entire memory access process consumes various resources such as cache hit check resources and bandwidth resources, resulting in resource waste. At the same time, because accessing a small amount of memory requires a lot of resources, it may cause back pressure upstream, leading to a decrease in cache performance.

[0054] In view of this, this application provides a cache request fusion method, which merges multiple data requests from the same address into a single data request for memory access, thereby reducing resource consumption and improving cache performance.

[0055] Before introducing the cache request fusion method in this application, let's first introduce a structural diagram of an artificial intelligence chip applicable to the embodiments of this application.

[0056] refer to Figure 1 The diagram shows the structure of an artificial intelligence chip applicable to the embodiments of this application. The artificial intelligence chip 100 includes at least a processing core 101, a cache 102, and main memory 103. The processing core 101 has the ability to perform artificial intelligence-related calculations, such as matrix multiplication and convolution operations, which are common calculation operations in artificial intelligence models.

[0057] Cache 102 is a small, fast storage area located between processing core 101 and main memory 103. It is used to temporarily store data or instructions that processing core 101 or other hardware devices may frequently access in the near future, with the aim of speeding up access and reducing access latency. Main memory 103 is the next level below cache 102. Compared to cache 102, main memory 103 generally has a larger capacity but is slower.

[0058] Cache 102 includes multi-level caching. The cache request fusion method in this application is applicable to LLC in multi-level caching, and also to other levels of caching. This application does not make specific limitations on this.

[0059] In some embodiments, see Figure 2 The cache 102 includes: a merge engine 201, an input buffer 202, a hit-miss check unit 203, a cache ram 204, and a schedule buffer 205.

[0060] In practice, the tag memory is used to store tag information for each cache line in multiple cache sets. A cache set is a logical group consisting of multiple cache lines. A cache line is the smallest unit of data storage in cache 102 and also the smallest unit of operation in cache 102.

[0061] A cache line includes data and tagging information. The tagging information includes a tag, a valid bit, and a dirty bit. The tag is used to identify the address range of the data in main memory. The valid bit indicates whether the data is valid. The dirty bit indicates whether the data has been modified, i.e., whether it is consistent with main memory.

[0062] The cache memory 204 stores the data in each cache line. The tag information stored in the tag memory has a one-to-one correspondence with the data stored in the cache memory 204.

[0063] In this embodiment, the merging engine 201 can receive data requests from multiple processing cores 101, or it can receive multiple data requests from a single processing core 101. Therefore, the first processing core and the second processing core described in this application are examples of processing core 101. The first processing core and the second processing core can refer to the same processing core, or they can refer to two independent processing cores.

[0064] Specifically, the merging engine 201 receives a first data request sent by the first processing core, the first data request including a first memory address. The merging engine 201 queries the incomplete buffer based on the first memory address. The incomplete buffer is used to store unfinished data requests received by the cache 102. When there is at least one second data request corresponding to the first memory address in the incomplete buffer, the first data request and at least one second data request are merged into a single target data request.

[0065] The target data request can be the earliest received second data request among at least one of the aforementioned second data requests. When the merging engine 201 receives the target data request, it has not completed the data request for which there is no corresponding first memory address in the buffer. Therefore, it forwards the target data request to the hit check unit 203. Second data requests and first data requests received after the target data request are not forwarded but are placed in a waiting state.

[0066] The hit checking unit 203 reads the tag information of each cache line from the tag memory; then performs a hit check on the first memory address of the target data request based on the read tag information, and sends the check result to the scheduling buffer 205.

[0067] When the first memory address matches the tag information of a cache line, the result of the check is: the target data request has hit a cache line; when no tag information matches the first memory address, the result of the check is: the target data request has not hit a cache line. Therefore, a cache line is allocated for the target data request, and the memory address of the cache line is written into the tag memory.

[0068] In some embodiments, if the first data request and the second data request are read requests corresponding to the same memory address, when the target data request hits a cache line, the scheduling buffer 205 schedules the reading of the cache line data from the cache memory 204 and returns the cache line data as a response to the target request to the merging engine 201; the cache line data may have been previously read from the main memory 103.

[0069] The merging engine 201 obtains the response content of the first data request and at least one second data request from the target request response; then returns the response content of the first data request to the first processing core; and returns the response content of each second data request to the corresponding second processing core.

[0070] After each response is returned to the corresponding processing core, the merging engine 201 releases the storage space used to save the target request response and sends a notification message to the scheduling buffer 205 that the storage space has been released. In this way, the scheduling buffer 205 can use the released storage space to save new target request responses.

[0071] In some embodiments, if the first data request and the second data request are write requests corresponding to the same memory address, the first data request further includes: an input buffer address and first data to be written. After receiving the first data request from the first processing core, the merging engine 201 writes the first data to be written into the second buffer in the input buffer 202 according to the input buffer address.

[0072] When a target data request hits a cache line, the scheduling buffer 205 schedules the reading of the full data already stored in the second buffer; the full data refers to all the data already stored in the second buffer; in practical applications, the full data not only includes the first data to be written, but may also include at least one second data to be written carried by each of the second data requests. The full data is then written to the storage area corresponding to that cache line in the cache memory 204.

[0073] In some cases, before writing all the data to the cache memory 204, an address lookup can be performed on the merging engine 201 to check the address of the input buffer where the data to be written is currently needed. After the lookup, the merging engine 201 will no longer allow newly received write requests corresponding to the first memory, and will merge them with the aforementioned first data request and at least one second data request.

[0074] After a successful write, the scheduling buffer 205 returns a write success message as a response to the target request to the merging engine 201. The merging engine 201 returns the write success message as the response content for the first data request to the first processing core, and returns the write success message as the response content for each second data request to the corresponding second processing core.

[0075] In some embodiments, see Figure 3 The merging engine 201 includes: a read allocation engine 301, a read incomplete list 302, a read data cache 303, a write allocation engine 304, a write incomplete list 305, a first selector 306, and a second selector 307.

[0076] Each record in the incomplete read list 302 includes the following elements: request identifier (req_id), memory address (addr), and pointer, where the pointer is used to point to the specific buffer in the read data cache 303.

[0077] Each message record in the incomplete write list 305 includes the following elements: request identifier (req_id), memory address (addr), and input buffer address (corresponding to the specific buffer in input buffer 202).

[0078] In some cases, the information records for reading incomplete list 302 and writing incomplete list 305 also include other elements such as sidebands.

[0079] When the first data request is a read request, the read allocation engine 301 receives the first data request and then adds a first information record of the first data request to the read incomplete list 302. The first information record includes: a first request identifier and a first memory address; the pointer in the first information record is a default pointer.

[0080] Based on the first memory address, scan the incomplete list 302; if at least one valid second data request corresponding to the first memory address is found, then the first data request will no longer be issued for execution, that is, the first data request will be set to a waiting state.

[0081] The first selector 306 receives the data of the cache line returned by the cache memory 204 (i.e., the data to be read in accordance with the target data request), and then saves the data of the cache line as the target request response in the first buffer of the read data cache 303. It queries the target information record of the target data request in the incomplete read list 302 and adjusts the default pointer in the target information record to point to the target pointer in the first buffer.

[0082] According to the target pointer and the memory access offset carried by the target data request, the response content of the target data request is read from the first buffer and returned to the corresponding second processing core; then, the target information record is set to invalid in the read incomplete list 302.

[0083] Scan the incomplete list 302 for data requests that correspond to the same memory address (i.e., the first memory address) as the target data request. These data requests include at least the first data request; in some cases, they may also include one or more second data requests.

[0084] For the first data request detected by scanning, the default pointer in the corresponding first information record is adjusted to point to the target pointer of the first buffer. According to the target pointer and the memory access offset carried by the first data request, the response content of the first data request is read from the first buffer and returned to the first processing core; then, the first information record is set to invalid in the read incomplete list 302.

[0085] For the second data request detected by scanning, the default pointer in the corresponding second information record is adjusted to point to the target pointer of the first buffer. According to the target pointer and the memory access offset carried by the second data request, the response content of the second data request is read from the first buffer and returned to the second processing core; then, the second information record is set to invalid in the read incomplete list 302.

[0086] After each response is returned to the corresponding processing core, the read data cache 303 releases the first buffer used to store the target request response and sends a reminder message that the first buffer has been released to the scheduling buffer 205. In this way, the scheduling buffer 205 can use the released first buffer to store new target request responses.

[0087] If the first data request is a write request, the write allocation engine 304 receives the first data request and then adds a first information record of the first data request to the write incomplete list 305. The first information record includes: a first request identifier, a first memory address, and the input buffer address of the second buffer. The first data to be written is written to the second buffer in the input buffer 202 according to the input buffer address.

[0088] Based on the first memory address, scan the incomplete write list 305; if at least one valid second data request corresponding to the first memory address is found, then the first data request will no longer be issued for execution, that is, the first data request will be set to a waiting state.

[0089] It should be noted that the incomplete write list 305 also includes a second information record for each second data request; both the first and second information records contain a first memory address and an input buffer address, and the request identifiers of the first and second information records are different. In at least one second data request, the second data request that has been issued and executed is the fused target data request (corresponding to the target information record).

[0090] The merging engine 201 receives the write success message (the write success message obtained from executing the target data request) sent by the scheduling buffer 205, and returns the write success message to the second processing core that sent the target data request; then it sets the target information record to invalid in the write incomplete list 305.

[0091] A write success message is returned to the first processing core corresponding to the first data request; then, the first information record is set to invalid in the write incomplete list 305.

[0092] A write success message is returned to the second processing core corresponding to the sending of other second data requests; then, the second information record is set to invalid in the write incomplete list 305.

[0093] Upon receiving the first data request, if no data request corresponding to the first memory address is found, the first data request is sent to the second selector 307 for judgment. When the second selector 307 selects to execute the first data request, it sends the first data request to the hit checking unit 203 for hit checking. At this time, the first data request is also the fused target data request. Data requests received after the first data request at the same address do not need to be sent but are in a waiting state. In this case, the processing procedure for the first data request is the same as the processing procedure for the target data request described above, and will not be repeated here.

[0094] In addition to the structure described above, the artificial intelligence chip 100 in this application may also include other structures, and this application does not specifically limit such structures.

[0095] Artificial intelligence chips 100 can be: Graphics Processing Unit (GPU), General-Purpose Graphics Processing Units (GPGPU), Domain Specific Architecture (DSA), etc.

[0096] The following is based on Figure 1 The diagram shown illustrates the architecture of an AI chip and details the process of a cached request fusion method. (See also...) Figure 4 This method is executed by a cache in an artificial intelligence chip and includes the following steps:

[0097] Step 401: Receive a first data request from the first processing core. The first data request includes a first memory address.

[0098] Specifically, the first data request can be any data request sent to the cache by the first processing core. The first memory address is used to perform a hit check on the first data request; the first data request can be a read request or a write request. The first data request also includes: a first request identifier, which is used to uniquely identify the first data request.

[0099] Step 402: When there is at least one second data request corresponding to the first memory address in the uncompleted buffer, merge the first data request and at least one second data request into a single target data request.

[0100] Specifically, the incomplete buffer is used to store cached but incomplete data requests. After receiving the first data request from the first processing core, a first information record of the first data request is added to the incomplete buffer. The first information record includes: a first request identifier and a first memory address.

[0101] Since the first data request and at least one second data request received by the cache correspond to the same memory address (i.e., the first memory address), the first data request and at least one second data request operate on the same cache line. If the first data request and at least one second data request are executed separately, the same cache line needs to be operated on repeatedly, which results in large resource overhead and low memory access efficiency.

[0102] Based on this, in this application, at least one of the second data requests that has been issued and executed is taken as the target data request; and the first data request is set to a waiting state.

[0103] When there are multiple second data requests corresponding to the first memory address, the target data request is the first second data request received among the multiple second data requests; when the target data request is received, no valid data request corresponding to the first memory address is found in the incomplete buffer.

[0104] Among the multiple second data requests, apart from the second data request that has already been issued and executed (i.e. the target data request), the other second data requests do not need to be issued, but are placed in a waiting state.

[0105] Step 403: Execute the target data request and obtain the target request response.

[0106] Specifically, the first data request and at least one second data request operate on the same cache line. Therefore, the target request response obtained by executing the target data request (i.e., a second data request) includes the response content of the first data request and at least one second data request, respectively.

[0107] In this way, the response content of the first data request is obtained from the target request response, and then the response content of the first data request is returned to the first processing core. The response content of each second data request is obtained from the target request response, and then the response content of each second data request is returned to the corresponding second processing core.

[0108] In this embodiment, the cache receives a first data request sent by the first processing core, and then queries the incomplete buffer based on the first memory address carried by the first data request. When at least one second data request with the same memory address is found, that is, when the first data request and at least one second data request need to operate on the same cache line, the first data request and at least one second data request are merged into a target data request. Then, a target data request is executed to obtain the response content of the first data request and at least one second data request respectively, thus reducing the number of data requests executed. Correspondingly, the number of times the resources necessary to execute data requests (such as hit check resources, bandwidth resources, and other resources) are consumed is reduced, thereby reducing resource waste and improving internal bandwidth.

[0109] Secondly, merging multiple data requests into a single data request increases the amount of data accessed in memory for a single request, thereby improving caching performance and memory access efficiency.

[0110] In addition, the cached request fusion method in this application only requires adding relevant logic when the data request enters the cache. The implementation logic is simple and does not increase the design difficulty of the cache, making it easy to implement.

[0111] In some embodiments, the incomplete buffer is a read incomplete list, and the first data request and the second data request are read requests. After receiving the first data request from the first processing core, a first information record of the first data request is added to the read incomplete list. The first information record includes: a first request identifier, a first memory address, and a default pointer.

[0112] The incomplete read list also includes a second information record for each second data request; both the first and second information records contain a first memory address and a default pointer; the request identifiers of the first and second information records are different. In at least one second data request, the second data request that has been issued and executed is the target data request (corresponding to the target information record).

[0113] For a target data request that has already been issued and executed, a hit check is first performed on the target data request. When the target data request hits a cache line, the data of the cache line is read from the cache memory based on the tag information of the cache line; then, the data of the cache line is added to the first buffer of the read data cache as the target request response.

[0114] In some embodiments, the size of the first buffer is the same as the size of a cache line, and the first buffer stores the data of one cache line. A target pointer to the first buffer is added to the target information record; then, the response content of the target data request is read from the first buffer according to the target pointer and the memory access offset carried by the target data request.

[0115] Specifically, in the target information record, the default pointer is adjusted to include a target pointer pointing to the first buffer. The target pointer is used to locate the first buffer from the read data cache; the memory access offset of the target data request is used to: locate the specific position of the target data request's response content in the first buffer, that is, the specific position of the target data request's response content within the target request response. The target data request's response content can be the target request response itself (i.e., the entire cache line of data read), or it can be a portion of the target request response (i.e., a portion of the data in the cache line).

[0116] In some embodiments, since the first data request and the target data request correspond to the same memory address (i.e., the first data request and the target data request are bound together), after adding a target pointer to the first buffer in the target information record, a target pointer to the first buffer is correspondingly added to the first information record. That is, in the first information record, the default pointer is adjusted to add a target pointer to the first buffer. Then, according to the target pointer and the memory access offset carried by the first data request, the response content of the first data request is read from the first buffer.

[0117] Specifically, the target pointer is used to locate the first buffer from the read data cache; the memory access offset carried by the first data request is used to: locate the specific position of the response content of the first data request in the first buffer, that is, the specific position of the response content of the first data request in the target request response. The response content of the first data request can be the entire cache line of data read, or a portion of the data in the cache line.

[0118] The response content of the first data request and the response content of the target data request can be completely different data or they can overlap. This application does not make specific limitations on this.

[0119] Furthermore, when there are multiple second data requests corresponding to the first memory address, in addition to the second data requests that have already been issued and executed (i.e., the target data requests), for each other second data request, a target pointer pointing to the first buffer is added to the second information record of the second data request; then, according to the target pointer and the memory access offset carried by the second data request, the response content of the second data request is read from the first buffer. The specific process will not be elaborated here.

[0120] The response content of the first data request and the response content of each of the multiple second data requests can be completely different data or they can overlap. This application does not make any specific limitations on this.

[0121] For example, the cache receives read request 1 (carrying memory address 1) sent by processing core 1, and adds information record 1 of read request 1 to the read incomplete list. Since no other information record containing memory address 1 is found in the read incomplete list, read request 1 is sent for execution.

[0122] If, during the execution of read request 1, a read request 2 (carrying memory address 1) is received from processing core 2, then information record 2 for read request 2 is added to the read incomplete list. Since information record 1 in the read incomplete list contains memory address 1, read request 2 is no longer sent for execution, but enters a waiting state.

[0123] During the execution of read request 1, a hit check is performed on read request 1. When read request 1 hits a cache line, the data of that cache line is read from the cache memory, where the cache line size is 512 bytes. The data of the cache line is added to the free buffer 1 in the read data cache.

[0124] Next, add pointer 1 to information record 1, pointing to buffer 1, and add pointer 1 to information record 2, pointing to buffer 1.

[0125] Set the memory access offset indicator carried by read request 1: offset 256 bytes from address 0. Based on this, locate buffer 1 in the read data cache according to pointer 1, and then read the first 256 bytes of data in buffer 1 as the response content of read request 1. Then return the response content of read request 1 to processing core 1, and at the same time, set information record 1 to invalid in the read incomplete list.

[0126] Set the memory access offset indicator carried by read request 2: offset 256 bytes from address 255. Based on this, locate buffer 1 in the read data cache according to pointer 1, and then read the last 256 bytes of data in buffer 1 as the response content of read request 2. Then return the response content of read request 2 to processing core 2, and at the same time, set information record 2 to invalid in the read incomplete list.

[0127] After both information record 1 and information record 2 are invalidated, buffer 1 is released; thus, buffer 1 can be used to store data of newly read cache lines.

[0128] In this embodiment, multiple read requests corresponding to the same memory address are merged into a single read request. Executing the merged read request yields the response content for each of the multiple read requests. Compared to executing multiple read requests separately, this reduces the number of resource consumption operations, such as hit checks and bandwidth usage, thereby reducing overall resource consumption. Furthermore, merging multiple read requests into a single read request effectively improves data reading efficiency, thus enhancing cache performance.

[0129] In some embodiments, the incomplete buffer is a list of incomplete writes; the first data request and the second data request are write requests; the first data request further includes: an input buffer address and a first data to be written.

[0130] After receiving the first data request from the first processing core, the first information record of the first data request is added to the write incomplete list. The first information record includes: the first request identifier, the first memory address, and the input buffer address.

[0131] The incomplete write list also includes a second information record for each second data request; both the first and second information records contain a first memory address and an input buffer address, and the request identifiers of the first and second information records are different. In at least one second data request, the second data request that has been issued and executed is the fused target data request (corresponding to the target information record).

[0132] After receiving the first data request from the first processing core, the first data to be written is written to the second buffer corresponding to the input buffer address.

[0133] Specifically, the size of the second buffer is the same as the size of a cache line. The first data to be written is less than or equal to the size of the second buffer. At this time, the first data to be written is written to the second buffer according to the input buffer address and the memory access offset carried by the first data request.

[0134] The input buffer address is used to locate the specific position of the second buffer within the input buffer; the memory access offset carried by the first data request is used to locate the specific writing position of the first data to be written in the second buffer.

[0135] In addition, the second buffer not only contains the first data to be written, but also the second data to be written carried by each second data request. The writing method for the second data to be written is the same as that for the first data to be written, and will not be described again here.

[0136] It should be noted that if multiple data requests write data to the second buffer sequentially and the write positions overlap, the data written later will overwrite the data written earlier.

[0137] For a target data request that has already been issued and executed, when the target data request hits a cache line, all the data already stored in the second buffer is read. The full data refers to all the data already stored in the second buffer; then, the full data is written to the storage area corresponding to the cache line in the cache memory; after a successful write, a write success message is sent as the target request response.

[0138] Specifically, the full data includes the first data to be written; in some cases, the full data may also include all or part of the data in the second data to be written. Through a single read / write operation, the full data in the second buffer is written to the storage area corresponding to the cache line in the cache memory.

[0139] Then, the write success message is returned to the first processing core as the response content of the first data request; and the write success message is returned to the corresponding second processing core as the response content of each second data request.

[0140] For example, the cache receives and processes write request 1 sent by core 1. Write request 1 includes: memory address 1, data to be written 1, input buffer address 1, and memory access offset 1. The size of data to be written 1 is 256 bytes; memory access offset 1 indicates an offset of 256 bytes from address 0; input buffer address 1 is the address of buffer 2 in the input buffer; the size of buffer 2 is 512 bytes.

[0141] Add write request 1 to the write incomplete list; then write the data to be written 1 to buffer 2 according to input buffer address 1 and memory access offset 1. Since no other record containing memory address 1 is found in the write incomplete list, write request 1 is sent for execution.

[0142] If, during the execution of write request 1, a write request 2 is received from processing core 2, write request 2 includes: memory address 1, data to be written 2, input buffer address 1, and memory access offset 2. The size of data to be written 2 is 256 bytes; memory access offset 2 indicates an offset of 256 bytes from address 255.

[0143] Add a record for write request 2 to the incomplete write list. Since record 1 in the incomplete write list contains memory address 1, write request 2 will not be issued for execution but will instead enter a waiting state. Write the data to be written 2 to buffer 2 according to input buffer address 1 and memory access offset 2. The data to be written 1 and the data to be written 2 will then fill buffer 2 completely.

[0144] During the execution of write request 1, a hit check is performed on write request 1. When write request 1 hits a cache line, the full amount of data already stored in buffer 2 (i.e., data to be written 1 and data to be written 2) is read; then the full amount of data is written to the storage area corresponding to the cache line in the cache memory.

[0145] After a successful write, a write success message is returned to processing core 1, and information record 1 is set to invalid in the write incomplete list. A write success message is then returned to processing core 2, and information record 2 is set to invalid in the write incomplete list.

[0146] In this embodiment, the data to be written for multiple write requests corresponding to the same memory address is stored in the same buffer. By merging these multiple write requests into a single write request, the data from the multiple write requests can be written to the cache memory all at once, without needing to perform hit checks and write to the cache memory separately for each write request. This effectively reduces the number of times resources such as hit check resources and bandwidth resources are consumed, thereby reducing overall resource consumption and improving internal bandwidth. Secondly, merging multiple write requests into a single write request effectively improves the efficiency of writing data, thus enhancing cache performance.

[0147] Based on the same technical concept, embodiments of this application provide a computer device, such as... Figure 5 As shown, it includes at least one artificial intelligence chip 100 and a memory 501 connected to at least one artificial intelligence chip 100. In this embodiment, the specific connection medium between the artificial intelligence chip 100 and the memory 501 is not limited. Figure 5 Taking the connection between the AI ​​chip 100 and the memory 501 via a bus as an example, the bus can be divided into address bus, data bus, control bus, etc.

[0148] In this embodiment, the memory 501 stores instructions that can be executed by at least one artificial intelligence chip 100. By executing the instructions stored in the memory 501, the at least one artificial intelligence chip 100 can perform the steps of the cached request fusion method described above.

[0149] The artificial intelligence chip 100 serves as the control center of the computer device. It connects to various parts of the computer device via various interfaces and lines, and achieves cached request fusion by running or executing instructions stored in the memory 501 and accessing data stored in the memory 501. Optionally, the artificial intelligence chip 100 may include one or more processing units. The artificial intelligence chip 100 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the artificial intelligence chip 100. In some embodiments, the artificial intelligence chip 100 and the memory 501 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0150] The artificial intelligence chip 100 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly manifested as being executed by a hardware processor, or executed by a combination of hardware and software modules within the processor.

[0151] Memory 501, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 501 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 501 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer device, but is not limited thereto. Memory 501 in the embodiments of this application may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0152] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program executable by a computer device, which, when run on the computer device, causes the computer device to perform the steps of the aforementioned cached request fusion method.

[0153] Those skilled in the art will understand that embodiments of the present invention can be provided as methods or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0154] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer apparatus or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0155] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer device or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0156] These computer program instructions may also be loaded onto a computer device or other programmable data processing equipment to cause a series of operational steps to be performed on the computer device or other programmable equipment to produce a process implemented by the computer device, thereby providing instructions that execute on the computer device or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0157] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0158] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method of fusing cached requests, the method comprising: The method comprises: receiving a first data request of a first processing core, the first data request comprising a first memory address; when there is at least one second data request corresponding to the first memory address in an uncompleted buffer, merging the first data request and the at least one second data request into a target data request, the uncompleted buffer being used to store the received and uncompleted data requests; executing the target data request to obtain a target request response; the target request response comprising the response content of the first data request and the at least one second data request respectively; when the first data request and the second data request are write requests, the first data request further comprises an input buffer address and first to-be-written data, and the second data request further comprises the input buffer address and second to-be-written data; when each second data request is received, writing the second to-be-written data into a second buffer according to the input buffer address and the memory access offset carried by the second data request; when the first data request is received, writing the first to-be-written data into the second buffer according to the input buffer address and the memory access offset carried by the first data request; the execution of the target data request to obtain a target request response comprises: when the target data request hits a cache line, reading the full amount of data stored in the second buffer; and writing the full amount of data into a storage area corresponding to the cache line in the cache memory; the full amount of data comprising all or part of the first to-be-written data and the second to-be-written data; after the writing is successful, a write success message is taken as the target request response.

2. The method of claim 1, wherein, the merging of the first data request and the at least one second data request into a target data request when there is at least one second data request corresponding to the first memory address in the uncompleted buffer comprises: when there is at least one second data request corresponding to the first memory address in the uncompleted buffer, taking one second data request that has been issued for execution in the at least one second data request as the target data request; placing the first data request in a waiting state.

3. The method of claim 1, wherein, the first data request further comprises a first request identifier; and after the first data request of the first processing core is received, the method further comprises: adding a first information record of the first data request in the uncompleted buffer, the first information record comprising the first request identifier and the first memory address.

4. The method of claim 1, wherein, after the execution of the target data request to obtain a target request response, the method further comprises: returning the response content of the first data request to the first processing core; and returning the response content of each second data request to a corresponding second processing core.

5. The method according to any one of claims 1 to 4, characterized in that, the uncompleted buffer is a read uncompleted list, and the first data request and the second data request are read requests; the execution of the target data request to obtain a target request response comprises: when the target data request hits a cache line, reading the data of the cache line from the cache memory based on the marking information of the cache line; and adding the data of the cache line as the target request response to a first buffer in a read data cache.

6. The method of claim 5, wherein, the target information record of the target data request is included in the read outstanding list; after the adding the data of the cache line as the target request response to a first buffer in a read data cache, the method further comprises: adding a target pointer pointing to the first buffer in the target information record; reading the response content of the target data request from the first buffer according to the target pointer and a memory access offset carried by the target data request.

7. The method of claim 5, wherein, the first information record of the first data request is included in the read outstanding list; after the adding the data of the cache line as the target request response to a first buffer in a read data cache, the method further comprises: adding a target pointer pointing to the first buffer in the first information record; reading the response content of the first data request from the first buffer according to the target pointer and a memory access offset carried by the first data request.

8. The method of claim 7, wherein, after the adding the write success message as the target request response after the write succeeds, the method further comprises: adding the write success message as the response content of each of the first data request and the at least one second data request.

9. An artificial intelligence chip, characterized by, comprises: a processing core, a cache and a main memory; the cache is configured to execute the method according to any one of claims 1 to 8.

Images