Memory merging device, data processing method, chip product and computer equipment

Abstract

The disclosure provides a memory merging device, a data processing method, a chip product and a computer device. In the memory merging device, a request merging module sends at least one read data request corresponding to a read data instruction to a cache module to obtain a current instruction state of the read data instruction; in a case where the current instruction state is a first state, the read data instruction is added to a first instruction queue, and in a case where the current instruction state is a second state, the read data instruction is added to a second instruction queue; a response selection module checks a current instruction state of a read data instruction at a head of the second instruction queue, and selects a target instruction queue from the first instruction queue and the second instruction queue; and a data return module returns target data of each read data request in a first read data instruction at a head of the target instruction queue to a requester of the first read data instruction after obtaining the target data from the cache module, so that the memory merging device can guarantee the stability of data access performance under a fast response mechanism.

Description

Technical Field

[0001] This disclosure relates to, but is not limited to, the field of chip technology, and in particular to a memory merging device, a data processing method, a chip product, and a computer device. Background Technology

[0002] In modern parallel computing chip designs such as GPUs (Graphics Processing Units) and AI (Artificial Intelligence), memory access efficiency plays a crucial role in overall system performance. To reduce the number of accesses to cache modules or downstream memory modules, memory merging technology can be used in GPU / AI chips. This involves merging multiple original requests at the cache line level to generate a request (also known as a cached request) adapted to the parallel processing capabilities of the cache module. Then, the memory merging device reads the request data of all cached requests corresponding to the original request from the cache module or downstream memory module to obtain the access data of the original request.

[0003] In related technologies, to meet the data access performance requirements of chips, memory merging devices typically incorporate a fast response mechanism for raw requests. Specifically, the memory merging device divides raw requests into two categories based on the hit status of their accessed data in the cache module, and shortens the processing path for each type of raw request to enable a fast response.

[0004] However, current memory merging devices may experience read data command response blocking issues under a fast response mechanism. To resolve this, the memory merging device needs to block the reception of read data commands, leading to an overall degrade in data access performance. Summary of the Invention

[0005] In view of this, the present disclosure provides at least one memory merging apparatus, a data processing method, a chip product, and a computer device.

[0006] The technical solution of this disclosure embodiment is implemented as follows: This disclosure provides a memory merging apparatus, including: The request merging module is configured to: based on a received read data instruction, send at least one read data request corresponding to the read data instruction to the cache module to obtain the current instruction status of the read data instruction; if the current instruction status is a first status, add the read data instruction to a first instruction queue; if the current instruction status is a second status, add the read data instruction to a second instruction queue; the first status indicates that the data ready status of each read data request corresponding to the read data instruction is ready, and the second status indicates that the data ready status of any read data request corresponding to the read data instruction is not ready; The response selection module is used to: check the current instruction status of the read data instruction at the head of the second instruction queue during the current clock cycle, and select a target instruction queue from the first instruction queue and the second instruction queue, wherein the current instruction status of the first read data instruction at the head of the target instruction queue is the first status; The data return module is used to: retrieve the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and return it to the requester of the first read data instruction.

[0007] This disclosure provides a data processing method applied to a memory merging apparatus, the memory merging apparatus including a request merging module, a response selection module, and a data return module; the method includes: The request merging module sends at least one read data request corresponding to the received read data instruction to the cache module to obtain the current instruction status of the read data instruction; if the current instruction status is a first status, the read data instruction is added to a first instruction queue; if the current instruction status is a second status, the read data instruction is added to a second instruction queue; the first status indicates that the data ready status of each read data request corresponding to the read data instruction is ready, and the second status indicates that the data ready status of any read data request corresponding to the read data instruction is not ready. Within the current clock cycle, the response selection module checks the current instruction status of the read data instruction at the head of the second instruction queue and selects a target instruction queue from the first instruction queue and the second instruction queue. The current instruction status of the first read data instruction at the head of the target instruction queue is the first status. The data return module retrieves the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and returns it to the requester of the first read data instruction.

[0008] This disclosure provides a chip product, which includes the memory merging device described in the above embodiments.

[0009] This disclosure provides a computer device, which includes the memory merging apparatus described in the above embodiments.

[0010] In this embodiment, the memory merging device includes a request merging module, a response selection module, and a data return module. The request merging module, based on a received read data instruction, sends at least one read data request corresponding to the read data instruction to the cache module, obtaining the current instruction state of the read data instruction. If the current instruction state is a first state, the read data instruction is added to a first instruction queue; if the current instruction state is a second state, the read data instruction is added to a second instruction queue. The response selection module, within the current clock cycle, checks the current instruction state of the read data instruction at the head of the second instruction queue and selects a target instruction queue from the first and second instruction queues, where the current instruction state of the read data instruction at the head of the target instruction queue is the first state. The data return module retrieves the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and returns it to the requester of the first read data instruction. The first state indicates that the data ready state of each read data request corresponding to the read data instruction is ready, and the second state indicates that the data ready state of any read data request corresponding to the read data instruction is not ready. The read-ready status indicates that the requested data has been hit in the cache module and can be read by the memory merging device.

[0011] In this way, the request merging module categorizes read data instructions into a first instruction queue and a second instruction queue based on whether all the data requested by each read data instruction is ready. This allows the response selection module to choose the target instruction queue from the two queues, ensuring that the first instruction queue, with all its read data requests already ready, is more likely to be selected than the second instruction queue. Consequently, the data return module reads and returns the data requested by the first read data instruction in the first instruction queue, effectively guaranteeing that read data instructions with ready data are returned to the requester more quickly. Furthermore, since the data requested by earlier read data instructions in the second instruction queue may also be ready, the second instruction queue also has a certain probability of being selected as the target instruction queue by the response selection module and processed by the data return module to return the requested data. Therefore, the technical solution disclosed herein can not only achieve a fast response to read data instructions that request data to be ready first, but also effectively ensure that other read data instructions can be responded to when the requested data is ready, reducing the probability of read data instruction blocking and ensuring the stability of data access performance of the memory merging device under the fast response mechanism.

[0012] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this disclosure. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the composition structure of a memory merging device provided in an embodiment of this disclosure. Figure 1 ; Figure 2 This is a schematic diagram of the composition structure of a memory merging device provided in an embodiment of this disclosure. Figure 2 ; Figure 3 This is a schematic diagram of the composition structure of a memory merging device provided in an embodiment of this disclosure. Figure 3 ; Figure 4 This is a schematic diagram of the structure of a second sub-queue provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of a first instruction queue provided in an embodiment of this disclosure; Figure 6 This is a schematic flowchart of a data processing method provided in an embodiment of this disclosure; Figure 7 This is a schematic diagram of the composition structure of a chip product provided in an embodiment of this disclosure; Figure 8 This is a schematic diagram of the composition structure of a computer device provided in an embodiment of this disclosure.

[0014] It should be noted that the terms "first" and "second" mentioned above are only used to distinguish between different options and do not represent the degree of superiority or inferiority of the options or their priority in the implementation process. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this disclosure clearer, the technical solutions of this disclosure are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this disclosure. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0016] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0017] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this disclosure described herein can be implemented in an order other than that illustrated or described herein.

[0018] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this disclosure.

[0019] The memory merging device is located between the upstream request module (such as the computing unit, graphics processing unit, controller, etc.) and the memory module. It is used to merge the original memory access requests corresponding to the memory access instructions sent by the upstream request module at the cache line granularity, resulting in at least one cache request. The memory access instructions can be, for example, read or write data instructions, and the cache requests can be, for example, read or write data requests. The merged cache request is then sent to the cache module or, through the cache module, to the downstream memory module for processing. Finally, the processing result is sent back to the upstream request module.

[0020] Among these, read data instructions such as data loading instructions and read-modify-write (RMW) instructions that require data to be returned, after the cache module / memory module returns the read data, the memory merging device can package the read data according to certain rules (data packing) and return it to the upstream request module. It should be noted that the memory merging device can also be called a memory merging system, memory merging module, memory merging unit, request merging system, request merging device, request merging unit, etc. Furthermore, in this embodiment, the read data instructions mainly refer to these types of read data instructions that require data to be returned.

[0021] In related technologies, each memory access instruction carries a warp identifier (Warp_id) in its instruction information. The warp_id is used to identify the warp to which the thread sending the memory access instruction belongs. In parallel computing on chips (such as GPU chips, AI chips, etc.), the warp is the basic unit of scheduling and execution in a stream multi-processor (SM).

[0022] For read data instructions that require data to be returned, there are generally three ways to return the data: all instructions are returned in the correct order (regardless of Warp_id), all instructions are returned in the correct order (regardless of Warp_id), and read data instructions are returned in the correct order by Warp_id (that is, read data instructions with the same Warp_id can return the corresponding read data in the correct order, and read data instructions with different Warp_ids can return the corresponding read data in the correct order).

[0023] In chip design, to optimize power-performance-area (PPA), the order of warp_ids is generally chosen. Assuming the chip system has a total of M warp_ids, in the hardware implementation, each warp_id typically corresponds to a logical or physical First-In-First-Out (FIFO) queue, referred to below as isched_fifo. The isched_fifo stores all read data instructions that fall under this warp_id. When all read requests corresponding to a read command are in the "Ready" state, the conditions for requesting command commit arbitration are met, and the command commit phase can begin. This allows the data return module to package and return the data requested by the read command. Similarly, when all read requests corresponding to the head of multiple `isched_fifo` queues are in the "Ready" state, one `isched_fifo` queue can be selected from the M `isched_fifo` queues for data return. In other words, a single command commit module (Commit arb) is needed to arbitrate among the M `isched_fifo` queues. The process of checking whether the read request corresponding to a read command is ready is performed by the data checking module. Typically, multiple data checking modules execute in parallel within a memory merging device, with each module corresponding to one of the multiple `isched_fifo` queues. In other words, each of the multiple `isched_fifo` queues has a data checking module to check whether all read requests corresponding to the read command at the head of the corresponding `isched_fifo` queue have ready data. The command submission module implements an arbitration logic to determine which read command's data can be returned to the upstream request module. Furthermore, it should be noted that the readiness status of a read request indicates whether the requested data is ready in the cache module, i.e., whether the requested data can be read immediately from the cache module. Specifically, a read request in a "ready" state means that valid data exists in the cache line corresponding to the read request, and the data can be read immediately from the cache module; a read request in a "not ready" state means that valid data does not exist in the cache line corresponding to the read request, and the data cannot be read immediately from the cache module. For example, a cache hit test can be performed based on the row address corresponding to the read request to obtain the hit test result, and the readiness status of the read request can be determined based on the hit test result.Specifically, if the cache hit result is a cache hit and valid data exists in the cache line corresponding to the specified row address, the read data request is in a ready state. If the cache hit result is a hit and valid data does not exist in the cache line corresponding to the specified row address (i.e., the cache module has requested the data corresponding to the specified row address from the downstream memory module, but has not yet obtained the data or has not yet written the data back to the cache line corresponding to the specified row address; this situation is also called a hit on miss, indicating that the data is in the loading process), the read data request is in a not ready state. If the cache miss result is a cache miss, the read data request is in a not ready state.

[0024] Therefore, if multiple upstream request modules connected to the same memory merging device support different data return methods—for example, the first upstream request module, requestor1, supports data return in warp_id order, while the second upstream request module, requestor2, supports data return in completely out-of-order order (ignoring warp_id)—then, in order to use the same hardware logic, the memory merging device will typically force the warp_id order data return method. This can lead to performance degradation of requestor2's read data commands in some scenarios.

[0025] For example, if all read requests corresponding to a read instruction from the second upstream request module (requestor2) are cache hits, but at least some read requests corresponding to a read instruction from the first upstream request module (requestor1) are cache misses, and these two read instructions have the same Warp_id, then these two read instructions will be written to the same isched_fifo queue. Since read instructions in the same isched_fifo queue must follow a first-in, first-out (FIFO) principle, the data requested by these two read instructions must be returned in the order they were written to the isched_fifo queue.

[0026] In particular, if the read data command of the second upstream request module requestor2 is written later than the read data command of the first upstream request module requestor1, then the read data command of the second upstream request module requestor2 will have to wait for the data requested by the read data command of the first upstream request module requestor1 to be returned before it can be processed and the requested data can be returned. Obviously, the data return mode supported by the read data command of the second upstream request module requestor2, which involves completely out-of-order data return for all commands, cannot realize its access performance.

[0027] In related technologies, to address this issue, for read data instructions that support instruction-level out-of-order scheduling (i.e., instructions can return corresponding read data in out-of-order order), such as read data instructions where the data return method involves all instructions being completely out of order, the memory merging device supports two data processing paths: a fast path and a slow path.

[0028] Specifically, in the memory merging device, the read instruction is entered into the fast path if all read requests corresponding to a single read instruction have access to the cached data. If all read requests have access to the cached data, the read instruction enters the slow path. If any read request has not accessed the cached data, the read instruction enters the slow path. In the slow path, after the read instruction is written to the corresponding `isched_fifo` queue according to its `Warp_id`, the data checking module checks whether all read requests corresponding to the read instruction are ready. If all read requests are ready, the read instruction is submitted to the instruction submission module, which then arbitrates and selects the `isched_fifo` queue from which data is to be returned before submitting it to the data return module. In the fast path, the read instruction can skip the data checking module and the instruction submission module and be submitted directly to the data return module. The data return module then arbitrates between the read instructions submitted through the fast path and those submitted through the instruction submission module, ultimately selecting one read instruction to return data. Clearly, under the fast path, read data commands do not need to pass through the data inspection module and the command submission module; they are at the same level as the output of the command submission module and participate in the arbitration of the data return module. In this way, read data commands under the fast path can reduce overall latency due to the shortened processing path, accelerate the response speed of read data commands, and improve data access performance.

[0029] However, the arbitration strategy for read data commands submitted via fast and slow paths in the data return module typically falls into two categories. One arbitration strategy assigns a fixed priority to read data commands submitted via fast paths, ensuring that the requested data from fast-path commands is returned to the upstream requesting module first. However, this strategy can lead to situations where slow-path read data commands fail to return, causing them to be continuously blocked due to not being able to participate in the arbitration. The other arbitration strategy does not assign priority to read data commands submitted via fast or slow paths, ensuring that both have the same arbitration probability. However, this strategy can also result in fast-path read data commands being continuously blocked due to not being able to participate in the arbitration.

[0030] To address the blocking issue of read data instructions, current memory merging devices typically employ a stall pipeline approach. This involves blocking the reception of new read data instructions while allowing existing ones to be processed sequentially. However, this results in an overall decrease in data access performance.

[0031] In view of this, embodiments of this disclosure provide a memory merging apparatus, such as... Figure 1 As shown, the memory merging device 10 includes: a request merging module 11, a response selection module 12, and a data return module 13. Wherein: The request merging module 11 is configured to: based on the received read data instruction, send at least one read data request corresponding to the read data instruction to the cache module 20 to obtain the current instruction status of the read data instruction; if the current instruction status is a first status, add the read data instruction to a first instruction queue; if the current instruction status is a second status, add the read data instruction to a second instruction queue. The first status indicates that the data ready status of each read data request corresponding to the read data instruction is ready, and the second status indicates that the data ready status of any read data request corresponding to the read data instruction is not ready.

[0032] The response selection module 12 is used to: check the current instruction status of the read data instruction at the head of the second instruction queue within the current clock cycle, and select the target instruction queue from the first instruction queue and the second instruction queue. The current instruction status of the first read data instruction at the head of the target instruction queue is the first status.

[0033] The data return module 13 is used to: retrieve the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module 20 and return it to the requester 30 of the first read data instruction.

[0034] Here, the upstream request module of the memory merging device 10 may include one or more requesters 30, and the requester of the read data instruction may be any one of the one or more requesters.

[0035] In some embodiments, after receiving a read data instruction, the request merging module 11 can determine at least one read data request corresponding to the received read data instruction and send each read data request to the cache module 20 to obtain the current instruction status of the read data instruction. The cache module 20 can be a module inside the memory merging device 10 or a module outside the memory merging device 10; this embodiment does not limit the specific module. Each read data instruction can correspond to at least one read data request, and the at least one read data request corresponding to a read data instruction can be used to read data from the cache module 20.

[0036] In some implementations, the read data instruction may carry the starting address and length of the data to be read. Based on the starting address and length, at least one read data request can be determined, and each read data request can be used to read data from a cache line in the cache module.

[0037] In some implementations, the request merging module 11 can merge at least one original request at the cache line granularity based on the data start address and data length corresponding to at least one original request in the read data instruction, to obtain at least one read data request. Each read data request can be used to read data within a cache line in the cache module. It is understood that the requester of the read data instruction can have a parallel execution unit. Multiple thread tasks executed by the parallel execution unit at one time can be called an instruction. If a thread task is used to read data from the cache module, the corresponding instruction is a read data instruction. This read data instruction is used to read data corresponding to at most b original requests from the cache module. Here, b is a positive integer. That is, the above b thread tasks correspond one-to-one with b original requests. The value of b can be preset, depending on the number of threads executing tasks in parallel in the parallel execution unit of the instruction requester, for example, b=4, 6, or 8, etc., which is not limited in this embodiment. It should be noted that b here is the maximum capability of an instruction; an instruction may have all b threads effective, or only a few of the b threads may be effective. Each original request has a corresponding data start address and data length. The data start address indicates the storage address of the data requested by the original request in the cache module; the data length indicates the length or amount of data requested by the original request.

[0038] The current instruction status of a read data instruction indicates whether the data readiness status of all read data requests in the read data instruction is ready.

[0039] In some embodiments, after receiving at least one read data request corresponding to a read data instruction, the request merging module 11 can send each read data request of the read data instruction to the cache module 20, obtain the data readiness status of each read data request, and determine the current instruction status of the read data instruction based on the data readiness status of each read data request. This current instruction status is either a first status or a second status. Furthermore, if the current instruction status of the read data instruction is in the first status, the request merging module 11 adds the read data instruction to a first instruction queue; if the current instruction status of the read data instruction is in the second status, it adds the read data instruction to a second instruction queue. In other embodiments, after sending each read data request of the read data instruction to the cache module 20, the request merging module 11 can obtain the current instruction status of the read data instruction, which includes the data readiness status of each read data request corresponding to the read data instruction. The request merging module 11 determines whether the data ready status of all read data requests is "ready". If the data ready status of all read data requests is "ready", indicating that the current instruction status of the read data instruction is the first state, the read data instruction is added to the first instruction queue. If the data ready status of any read data request is "not ready", indicating that the current instruction status of the read data instruction is the second state, the read data instruction is added to the second instruction queue.

[0040] In some embodiments, each read data request is used to read data from a cache line in the cache module 20. The request merging module 11 can send each read data request corresponding to the read data instruction to the cache module 20 to detect the data readiness status of each read data request based on the line address corresponding to each read data request, and obtain the current instruction status of the read data instruction.

[0041] In this context, the starting address of at least one read data request corresponding to a read data instruction is a consecutive set of at least one row address. Therefore, at least one read data request corresponding to this read data instruction can be used to read data from at least one cached row with consecutive row addresses. For example, the read data instruction Instruction1 corresponds to read data requests cache_request1, cache_request2, and cache_request3. The starting addresses of the data in read data requests cache_request1, cache_request2, and cache_request3 are the row addresses cacheline_id1, cacheline_id2, and cacheline_id3, respectively, where the row addresses cacheline_id1, cacheline_id2, and cacheline_id3 are consecutive.

[0042] In an optional embodiment, the request merging module 11 may employ any suitable parallel checking logic within a single clock cycle to check the data readiness status of all read data requests corresponding to the read data instruction in parallel. This embodiment of the present disclosure does not limit this. For example, the request merging module 11 may be configured with multiple parallel checking channels, each checking channel being used to check the data readiness status of a read data request based on the row address corresponding to that read data request.

[0043] Since the data requested by multiple read data requests corresponding to a single read data instruction is typically stored in multiple consecutive cache lines, meaning these multiple read data requests are used to read data from at least one cache line with consecutive row addresses, and the data requested by each read data request corresponding to the same read data instruction is usually ready within a few close clock cycles, checking the data readiness status of each read data request in parallel based on the row address corresponding to each read data request of the received read data instruction can improve the resource utilization of the request merging module 11 and the efficiency of checking the data readiness status of each read data request corresponding to the read data instruction, thereby further improving the overall performance of the memory merging device.

[0044] Alternatively, the request merging module 11 can check the data readiness status of all read data requests corresponding to the read data instruction over multiple clock cycles. Accordingly, the request merging module 11 can check the data readiness status of a first number of read data requests in parallel within a single clock cycle, so as to achieve the check of the data readiness status of all read data requests corresponding to the read data instruction over multiple clock cycles.

[0045] After receiving a read data instruction, the request merging module 11 can determine the row addresses of a first number of read data requests to be checked in the current clock cycle based on the row addresses of at least one read data request corresponding to the read data instruction. Within the current clock cycle, based on the row addresses corresponding to each of the first number of read data requests, the data readiness status of the first number of read data requests is checked in parallel. For example, the request merging module 11 can use the smallest row address among the data requests corresponding to the read data instruction as the starting row address to be checked, and determine the first number of row addresses starting from this starting row address as the row addresses to be checked in the current clock cycle.

[0046] In some embodiments, each cache line in the cache module 20 has a status identifier indicating whether the data in the cache line is ready. The request merging module 11 can also send each read data request corresponding to the read data instruction to the cache module 20, so that for each read data request, based on the row address corresponding to the read data request, if the cache module 20 does not hit the read data request, the data ready status of the read data request is determined to be not ready; if the cache module 20 hits the read data request, the status identifier of the target cache line corresponding to the read data request is queried, and the data ready status of the read data request is determined based on the status identifier of the target cache line.

[0047] Here, after the request merging module 11 sends each read data request to the cache module 20, the cache module 20 can perform a cache hit check on the received read data requests. This allows the request merging module 11 to receive the hit detection results of the cache module 20 for each read data request, determine that the read data requests that did not hit according to the hit detection results are in a non-ready state, and query the status identifier of the target cache line corresponding to each read data request that hit according to the hit detection results. Based on the status identifier of the target cache line, the read data ready state of the read data request is determined.

[0048] The caching module 20 can maintain a status flag for each cache line to indicate whether the data in that cache line is ready. Thus, the request merging module 11 can determine the data readiness status of each target read data request by querying the status flag of the target cache line corresponding to the read data request, for each read data request hit by the caching module 20.

[0049] For example, please continue to refer to Figure 1 When the cache module 20 receives a read data request, it can perform a cache hit check based on the row address corresponding to the read data request. If valid data exists in the cache row corresponding to the row address (i.e., a hit), the status flag of the cache row is set to a first flag indicating that the data in the cache row is ready. If no valid data exists in the cache row corresponding to the row address (i.e., a miss), the status flag of the cache row is set to a second flag indicating that the data in the cache row is not ready. Then, the cache module 20 requests the data corresponding to the read data request from the downstream memory module. Furthermore, when the data is written back to the cache row corresponding to the row address in the cache module, the status flag of the cache row is updated to the first flag.

[0050] In the above embodiments, each cache line in the cache module 20 has a status identifier indicating whether the data in that cache line is ready. The request merging module 11, in parallel, queries the status identifier of the target cache line corresponding to each read data request corresponding to the read data instruction, based on the row address corresponding to the read data request, and determines the data ready status of each read data request based on the status identifier of the target cache line corresponding to each read data request. In this way, within a single clock cycle, the data ready status of each read data request corresponding to the read data instruction can be checked in parallel based on the row address corresponding to each read data request, and the current instruction status of the read data instruction can be obtained.

[0051] In some embodiments, the first instruction queue may be a FIFO queue, and / or the second instruction queue may be a FIFO queue.

[0052] In some embodiments, the request merging module 11 is further configured to: add the read data instruction to the first instruction queue when the current instruction state of the read data instruction is the first state and the read data instruction supports instruction-level out-of-order scheduling.

[0053] Here, instruction-level out-of-order scheduling refers to read data instructions supporting out-of-order return regardless of Warp_id constraints. In other words, read data instructions with different Warp_ids can return the corresponding read data in out-of-order order. For example, read data instructions supporting instruction-level out-of-order scheduling could mean that they support the second data return method mentioned above (ignoring Warp_id, all instructions are completely out of order).

[0054] In some embodiments, the request merging module 11 can determine whether a read data instruction supports instruction-level out-of-order scheduling when the current instruction state of the read data instruction is in a first state. If the read data instruction supports instruction-level out-of-order scheduling, the read data instruction is added to the first instruction queue; if the read data instruction does not support instruction-level out-of-order scheduling, the read data instruction is added to the second instruction queue. Specifically, when the second instruction queue includes multiple first sub-queues, or when the second instruction queue includes multiple first sub-queues and second sub-queues, the request merging module 11 adds the read data instruction to the first sub-queue corresponding to the thread bundle if the read data instruction does not support instruction-level out-of-order scheduling. For example, the read data request includes its supported data return methods. The request merging module 11 adds the read data instruction to the first instruction queue if the current instruction state of the read data instruction is in the first state and the read data instruction supports a data return method where all instructions are completely out of order.

[0055] Clearly, read data instructions that can be written into the first instruction queue must satisfy the following conditions: all read data requests corresponding to the read data request must be in the ready state, and the read data instructions must support instruction-level out-of-order scheduling. Since the first instruction queue does not correspond to a thread bundle, read data instructions in the first instruction queue may not be returned according to the thread bundle's classification of read data instructions written within it. Therefore, read data instructions in the first instruction queue need to support instruction-level out-of-order scheduling to reduce the erroneous return of read data instructions that do not support instruction-level out-of-order scheduling, and lower the probability of processing errors by the memory merging device.

[0056] Within the current clock cycle, the response selection module 12 checks the current instruction status of the read data instruction at the head of the second instruction queue and selects the target instruction queue from the first and second instruction queues. The current instruction status of the first read data instruction at the head of the target instruction queue is the first state.

[0057] In some implementations, the response selection module 12 can cyclically check the data readiness status of each read data request corresponding to the read data instruction at the head of the second instruction queue in the cache module 20 within multiple clock cycles to obtain the current instruction status of the read data instruction.

[0058] In an optional configuration, the response selection module 12 may perform a check on the current instruction status of the read data instruction at the head of the second instruction queue once per clock cycle. That is, the duration of a single check on the current instruction status of the read data instruction at the head of the second instruction queue can be one clock cycle. The response selection module 12 may employ any suitable parallel checking logic within a single clock cycle to check the data readiness status of each read data request of the read data instruction at the head of the second instruction queue in parallel, thereby obtaining the current instruction status of that read data instruction. This embodiment of the present disclosure does not limit this approach. For example, the request merging module 11 may be configured with multiple parallel checking channels, each channel used to check the data readiness status of one read data request.

[0059] In another optional configuration, the response selection module 12 can complete a check of the current instruction status of the read data instruction at the head of the second instruction queue over multiple clock cycles. That is, the duration of a single check of the current instruction status of the read data instruction at the head of the second instruction queue can be multiple clock cycles. Within each clock cycle, the response selection module 12 can select at least one first read data request from all read data requests corresponding to the read data instruction at the head of the second instruction queue, and check the data readiness status of each first read data request in the cache module 20. Given the data readiness status of all read data requests corresponding to the read data instruction, the module 12 can then determine whether the data readiness status of all read data requests corresponding to the read data instruction is ready. Therefore, if all read data requests are in a ready state, the current instruction state of the read data instruction is determined to be in the first state; if the read data request is in a not ready state, the current instruction state of the read data instruction remains in the second state. This completes one check of the current instruction state of the read data instruction at the head of the second instruction queue, and so on, to achieve a cyclical check of the current instruction state of the read data instruction at the head of the second instruction queue. The first read data request is defined as a read data request whose ready state has not been checked in the current round of checks on the current instruction state of the read data instruction.

[0060] For example, the read data instruction at the head of the second instruction queue includes four read data requests: request 1, request 2, request 3, and request 4. The response selection module 12 can select request 1 and request 2 as the first read data requests within the first clock cycle and send each first read data request to the cache module 20 to obtain the data readiness status of each first read data request. Furthermore, the response selection module 12 can select the remaining request 3 and request 4 as the first read data requests within the second clock cycle and send each first read data request to the cache module 20 to obtain the data readiness status of each first read data request. If the data readiness status of all four read data requests is ready, the response selection module 12 determines the current instruction status of the read data instruction as the first state; if the data readiness status of any read data request is not ready, the current instruction status of the read data instruction remains in the second state, thus completing one check of the current instruction status of the read data instruction at the head of the second instruction queue. This process is repeated cyclically to check the current instruction status of the read data instruction at the head of the second instruction queue.

[0061] In some embodiments, each read data request is used to read data from a cache line in the cache module 20. The response selection module 12 can send each read data request corresponding to the read data instruction at the head of the second instruction queue to the cache module 20 to detect the data readiness status of each read data request based on the row address corresponding to each read data request. In other embodiments, each cache line in the cache module 20 has a status identifier for characterizing whether the data in the cache line is ready. The response selection module 12 can also send each read data request corresponding to the read data instruction at the head of the second instruction queue to the cache module 20 to determine the data readiness status of the read data request as not ready if the cache module 20 does not hit the read data request, based on the row address corresponding to the read data request; if the cache module 20 hits the read data request, it queries the status identifier of the target cache line corresponding to the read data request and determines the data readiness status of the read data request based on the status identifier of the target cache line. The specific implementation method of the response selection module 12 checking the data readiness status of each read data request can be referred to the specific implementation method of the request merging module 11 checking the data readiness status of each read data request corresponding to the read data instruction. This application will not elaborate on this.

[0062] The response selection module 12 can also select the target instruction queue for the current clock cycle from the first instruction queue and the second instruction queue according to the target arbitration strategy in each clock cycle, so that the data return module 13 can retrieve the target data requested by each read data request in the first read data instruction at the head of the target instruction queue from the cache module 20 and return it to the requester of the first read data instruction in the current clock cycle.

[0063] In some embodiments, the target arbitration strategy is a random strategy. The response selection module 12 can select any one of the first and second instruction queues whose current instruction state of the read data instruction at the head of the queue is in the first state as the target instruction queue. It is easy to understand that since the write condition of the read data instruction in the first instruction queue is that the current instruction state is in the first state, that is, the current instruction state of each read data instruction in the first instruction queue is in the first state, then when the current instruction state of the read data instruction at the head of the second instruction queue is in the first state, the second instruction queue and the first instruction queue have the same probability of being selected as the target instruction queue.

[0064] Clearly, since all read data instructions in the first instruction queue are currently in the first instruction state, the first instruction queue has a probability of being selected as the target instruction queue in each clock cycle. The second instruction queue, however, has a probability of not being selected as the target instruction queue within a single clock cycle because the read data instruction at its head may be in the second instruction state. Therefore, overall, the first instruction queue has a higher probability of being selected as the target instruction queue compared to the second instruction queue, effectively ensuring that read data instructions in the first instruction queue that request data are ready first can return data to the requester more quickly. Furthermore, since the read data instruction at its head in the second instruction queue may also be in the first instruction state, it also has a probability of being selected as the target instruction queue within a single clock cycle, and thus processed by the data return module to return the requested data. Based on this, the technical solution disclosed herein can, while ensuring a fast response to read data instructions in the first instruction queue that request data to be ready first, also effectively guarantee that other read data instructions in the second instruction queue can be responded to when their requested data is ready. This reduces the probability of read data instruction blocking and the possibility of pipeline pauses due to read data instruction blocking, thus ensuring the stability of data access performance of the memory merging device under the fast response mechanism. Compared to related technologies where the data return module uses an arbitration strategy that prioritizes read data instructions submitted via the fast path over those submitted via the slow path, this technical solution can, to a certain extent, effectively prevent read data instructions in the second instruction queue that have not yet been ready first (equivalent to read data instructions entering the slow path processing) from being continuously blocked due to their lower priority and inability to compete for arbitration.

[0065] In other embodiments, the target arbitration strategy is a polling strategy. The response selection module 12 can sequentially select one of the first and second instruction queues as candidate instruction queues according to a fixed polling order, and determine whether the current instruction state of the read data instruction at the head of the candidate instruction queue is in the first state. If the current instruction state of the read data instruction at the head of the candidate instruction queue is in the first state, the candidate instruction queue is determined as the target instruction queue. If the current instruction state of the read data instruction at the head of the candidate instruction queue is in the second state, the candidate instruction queue is updated according to the fixed polling order, and the current instruction state of the read data instruction at the head of the updated candidate instruction queue is determined to be in the first state. This process continues until the target instruction queue is determined.

[0066] Clearly, since all read data instructions in the first instruction queue are currently in the first instruction state, when the response selection module 12 polls and arbitrates the first instruction queue, it can be guaranteed to be determined as the target instruction queue. The second instruction queue, however, has a possibility that its head read data instruction might be in the second instruction state, meaning it might not be selected as the target instruction queue when polled by the response selection module 12. Therefore, overall, the first instruction queue has a higher probability of being selected as the target instruction queue compared to the second instruction queue, effectively ensuring that read data instructions in the first instruction queue that request data are ready first can return data to the requesting party more quickly. Furthermore, since the head read data instruction in the second instruction queue might also be in the first instruction state when polled by the response selection module 12, it also has a probability of being selected as the target instruction queue within a single clock cycle, thus being processed by the data return module to return the requested data.

[0067] Based on this, the technical solution disclosed herein can not only achieve a fast response to read data instructions in the first instruction queue that request data to be ready first, but also effectively ensure that other read data instructions in the second instruction queue can be responded to when their requested data is ready. This reduces the probability of read data instruction blocking and the possibility of pipeline pauses due to read data instruction blocking, thus ensuring the stability of data access performance of the memory merging device under the fast response mechanism. Compared with related technologies where the data return module processes read data instructions according to an arbitration strategy, the technical solution disclosed herein can not only, to a certain extent, prevent read data instructions in the second instruction queue that request data to be ready first (equivalent to read data instructions entering the slow path processing) from being continuously blocked due to their low priority and inability to compete for arbitration, but also effectively guarantee the arbitration right of read data requests in the first instruction queue (equivalent to read data instructions entering the fast path processing). Furthermore, when challenged, the requested data of the read data request can be returned, ensuring a fast response to read data instructions in the first instruction queue that request data to be ready first.

[0068] In this embodiment, the memory merging device includes a request merging module, a response selection module, and a data return module. The request merging module, based on a received read data instruction, sends at least one read data request corresponding to the read data instruction to the cache module, obtaining the current instruction state of the read data instruction. If the current instruction state is a first state, the read data instruction is added to a first instruction queue; if the current instruction state is a second state, the read data instruction is added to a second instruction queue. The response selection module, within the current clock cycle, checks the current instruction state of the read data instruction at the head of the second instruction queue and selects a target instruction queue from the first and second instruction queues, where the current instruction state of the read data instruction at the head of the target instruction queue is the first state. The data return module retrieves the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and returns it to the requester of the first read data instruction. The first state indicates that the data ready state of each read data request corresponding to the read data instruction is ready, and the second state indicates that the data ready state of any read data request corresponding to the read data instruction is not ready. The read-ready status indicates that the requested data has been hit in the cache module and can be read by the memory merging device.

[0069] In this way, the request merging module categorizes read data instructions into a first instruction queue and a second instruction queue based on whether all the data requested by each read data instruction is ready. This allows the response selection module to choose the target instruction queue from the two queues, ensuring that the first instruction queue, with all its read data requests already ready, is more likely to be selected than the second instruction queue. Consequently, the data return module reads and returns the data requested by the first read data instruction in the first instruction queue, effectively guaranteeing that read data instructions with ready data are returned to the requester more quickly. Furthermore, since the data requested by earlier read data instructions in the second instruction queue may also be ready, the second instruction queue also has a certain probability of being selected as the target instruction queue by the response selection module and processed by the data return module to return the requested data. Therefore, the technical solution disclosed herein can not only achieve a fast response to read data instructions that request data to be ready first, but also effectively ensure that other read data instructions can be responded to when the requested data is ready, reducing the probability of read data instruction blocking and ensuring the stability of data access performance of the memory merging device under the fast response mechanism.

[0070] In this implementation, the second instruction queue can exist in the following two situations, which will be explained in subsequent classifications.

[0071] In the first alternative case, such as Figure 2 As shown, the second instruction queue includes multiple first sub-queues.

[0072] The request merging module 11 is further configured to: determine the first thread bundle to which the thread sending the read data instruction belongs when the current instruction state of the read data instruction is the second state, and add the read data instruction to the first sub-queue corresponding to the first thread bundle.

[0073] Here, the first sub-queue has a corresponding relationship with the thread bundle. For example, the correspondence between the first sub-queue and the thread bundle can be one-to-one, one-to-many, or many-to-one. That is, each sub-queue corresponds to one or more thread bundles. Each first sub-queue can be used to store at least one read data instruction processed by each thread in the thread bundle corresponding to the first sub-queue. In some embodiments, each first sub-queue is used to store instruction request information of read data instructions processed by each thread in the thread bundle corresponding to the first sub-queue. The request merging module 11 can add the instruction request information of the read data instruction to the first sub-queue corresponding to the thread bundle where the read data instruction is located when the current instruction state of the read data instruction is the second state. The instruction request information of the read data instruction may include, but is not limited to, at least one of the following: the first identification information of the read data instruction, the second identification information of at least one read data request corresponding to the read data instruction.

[0074] In some implementations, the first subqueue is a first-in-first-out (FIFO) queue, and / or, the first instruction queue is a FIFO queue.

[0075] In some implementations, the read data instruction carries a thread bundle identifier, which identifies the thread bundle to which the thread sending the read data instruction belongs. Each thread bundle corresponds to a first sub-queue, and a single first sub-queue is used to store instruction request information for at least one read data instruction submitted by threads in the same thread bundle. The request merging module 11 can be used to: determine the first thread bundle to which the thread sending the read data instruction belongs when the current instruction state of the read data instruction is the second state; determine the instruction request information of the read data instruction based on the first identifier information of the read data instruction and the second identifier information of each read data request corresponding to the read data instruction; and store the instruction request information of the read data instruction in the instruction queue corresponding to the first thread bundle. In this way, since read data instructions submitted by threads in the same thread bundle are stored in the same instruction queue, read data instructions corresponding to the same thread bundle can return the corresponding request data in order, while read data instructions from different thread bundles can return the corresponding request data out of order.

[0076] In some embodiments, please refer to Figure 2The response selection module 12 includes: multiple first data checking modules 121, and an instruction submission module 122. Wherein: Among the multiple first data checking modules 121, the first data checking module 121 has a one-to-one or one-to-many relationship with the first sub-queue. The first data checking module 121 is used to: check the current instruction status of the read data instruction at the head of the corresponding first sub-queue within the current clock cycle.

[0077] The instruction submission module 122 is configured to: select a target instruction queue from a first instruction queue and at least one candidate sub-queue within the current clock cycle. The at least one candidate sub-queue is a sub-queue whose current instruction state is a first state for the read data instruction at the head of one of the multiple first sub-queues.

[0078] In some embodiments, each first data checking module 121 may correspond to one or more first sub-queues. The first data checking module 121 may cyclically check the data readiness status of each read data request of the second read data instruction at the head of the first sub-queue corresponding to the first data checking module 121 in the cache module 20 within multiple clock cycles to obtain the current instruction status of the second read data instruction.

[0079] In an optional configuration, the first data checking module 121 may perform a check on the current instruction status of the second read data instruction at the head of its corresponding first sub-queue once per clock cycle. That is, the duration of a single check on the current instruction status of the second read data instruction can be one clock cycle. The first data checking module 121 may employ any suitable parallel checking logic within a single clock cycle to check the data readiness status of each read data request of the second read data instruction at the head of the corresponding first sub-queue in parallel, thereby obtaining the current instruction status of the second read data instruction. This embodiment of the present disclosure does not limit this approach.

[0080] In another optional configuration, the first data checking module 121 can complete a check of the current instruction status of the second read data instruction at the head of the corresponding first sub-queue over multiple clock cycles. That is, the duration of a single check of the current instruction status of the second read data instruction can be multiple clock cycles. Within each clock cycle, the first data checking module 121 can select at least one second read data request from all read data requests of the second read data instruction at the head of the first sub-queue corresponding to it, and check the data readiness status of each second read data request in the cache module 20. Based on the obtained data readiness statuses of all read data requests corresponding to the second read data instruction, it can then determine whether the data readiness status of all read data requests corresponding to the second read data instruction is ready. Therefore, if the data ready status of all read data requests corresponding to the second read data instruction is "ready", the current instruction status of the second read data instruction is determined to be the first state; if the data ready status of any read data request is "not ready", the current instruction status of the second read data instruction remains in the second state, completing one check of the current instruction status of the second read data instruction at the head of the first sub-queue corresponding to the first data check module 121. This process is repeated to achieve a cyclical check of the current instruction status of the second read data instruction at the head of the first sub-queue corresponding to the first data check module 121. The second data request is defined as a read data request whose data ready status has not been checked in the current round of checks on the current instruction status of the second read data instruction.

[0081] In some embodiments, each read data request is used to read data from a cache line in the cache module 20. The first data checking module 121 can send the read data request corresponding to the second read data instruction at the head of the second instruction queue to the cache module 20 to detect the data readiness status of the read data request based on the row address corresponding to the read data request. In other embodiments, each cache line in the cache module 20 has a status identifier to characterize whether the data in the cache line is ready. The first data checking module 121 can also send the read data request corresponding to the second read data instruction at the head of the second instruction queue to the cache module 20 to determine the data readiness status of the read data request as not ready if the cache module 20 does not hit the read data request, based on the row address corresponding to the read data request; if the cache module 20 hits the read data request, it queries the status identifier of the target cache line corresponding to the read data request and determines the data readiness status of the read data request based on the status identifier of the target cache line. The specific implementation method of the first data inspection module 121 checking the data readiness status of the read data request can be referred to the specific implementation method of the aforementioned request merging module 11 checking the data readiness status of the read data request, and will not be elaborated here.

[0082] For example, the second instruction queue includes p first sub-queues slow path isched_fifo1-slow pathisched_fifo p. The response selection module 12 includes p first data check modules 121_1-121_p, each corresponding to a first sub-queue. The first data check module 121_i can check the data readiness status of each read data request in the second read data instruction at the head of the first sub-queue slow pathisched_fifo i in each clock cycle, to determine the current instruction status of the second read data instruction based on the data readiness status of each read data request. Here, i is a positive integer less than p.

[0083] The instruction submission module 122 can select the target instruction queue for the current clock cycle from at least one candidate sub-queue in the first instruction queue and the first sub-queue according to the target arbitration strategy in each clock cycle, so that the data return module 13 can retrieve the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module 20 and return it to the requester of the first read data instruction in the current clock cycle.

[0084] In some embodiments, the target arbitration strategy is a random strategy. The instruction submission module 122 can select the first sub-queue whose current instruction state of the read data instruction at the head of the queue is in the first state as a candidate sub-queue, and then select any instruction queue from the first instruction queue and the candidate sub-queues as the target instruction queue. It is easy to understand that since the write condition of the read data instruction in the first instruction queue is that the current instruction state is in the first state, that is, the current instruction state of each read data instruction in the first instruction queue is in the first state, the probability of the first instruction queue and the candidate sub-queue being selected as the target instruction queue is the same.

[0085] Clearly, since all read data instructions in the first instruction queue are currently in the first instruction state, the first instruction queue has a probability of being selected as the target instruction queue in each clock cycle. However, the first sub-queue, because the read data instruction at its head may be in the second instruction state, has a probability of not being selected as the target instruction queue in a single clock cycle. Therefore, overall, the first instruction queue has a higher probability of being selected as the target instruction queue than the first sub-queue, effectively ensuring that read data instructions in the first instruction queue that request data are ready first can return data to the requester more quickly. Furthermore, since the read data instruction at its head in the first sub-queue may also be in the first instruction state, it also has a probability of being selected as the target instruction queue in a single clock cycle, and thus processed by the data return module to return the requested data. Based on this, the technical solution disclosed herein can, while ensuring a fast response to read data instructions that request data to be ready first in the first instruction queue, also effectively guarantee that other read data instructions in the first sub-queue can be responded to when their requested data is ready. This reduces the probability of read data instruction blocking and the possibility of pipeline pauses due to read data instruction blocking, thus ensuring the stability of data access performance of the memory merging device under the fast response mechanism. Compared to related technologies where the data return module uses an arbitration strategy that prioritizes read data instructions submitted via the fast path over those submitted via the slow path, this technical solution can, to a certain extent, prevent read data instructions in the first instruction queue that have not yet been ready (equivalent to read data instructions entering the slow path) from being continuously blocked due to their lower priority and inability to compete for arbitration.

[0086] In other embodiments, the target arbitration strategy is a polling strategy. The instruction submission module 122 can sequentially select one instruction queue from the first instruction queue and one instruction queue from each of the first sub-queues as candidate instruction queues according to a fixed polling order, and determine whether the current instruction state of the read data instruction at the head of the candidate instruction queue is in the first state. If the current instruction state of the read data instruction at the head of the candidate instruction queue is in the first state, the candidate instruction queue is determined as the target instruction queue. If the current instruction state of the read data instruction at the head of the candidate instruction queue is in the second state, the candidate instruction queue is updated according to a fixed polling order, and the current instruction state of the read data instruction at the head of the updated candidate instruction queue is determined to be in the first state. This process continues until the target instruction queue is determined.

[0087] Clearly, since all read data instructions in the first instruction queue are currently in the first instruction state, when the instruction submission module 122 polls and arbitrates the first instruction queue, it can be guaranteed to be identified as the target instruction queue. The first sub-queue, however, has a possibility that its head read data instruction might be in the second instruction state, meaning it might not be selected as the target instruction queue when polled and arbitrated by the response selection module 12. Therefore, overall, the first instruction queue has a higher probability of being selected as the target instruction queue compared to the first sub-queue, effectively ensuring that read data instructions in the first instruction queue that request data are ready first can return data to the requester more quickly. Furthermore, since the head read data instruction of the first sub-queue might also be in the first instruction state when polled and arbitrated by the instruction submission module 122, it also has a probability of being selected as the target instruction queue within a single clock cycle, thus being processed by the data return module to return the requested data.

[0088] Based on this, the technical solution disclosed herein can not only achieve a fast response to read data instructions in the first instruction queue that request data to be ready first, but also effectively ensure that other read data instructions in the first sub-queue can be responded to when the requested data is ready, reducing the probability of read data instruction blocking and the possibility of pipeline pauses due to read data instruction blocking, thus ensuring the stability of data access performance of the memory merging device under the fast response mechanism. Compared with the related technology where the data return module processes read data instructions according to an arbitration strategy, the technical solution disclosed herein can not only, to a certain extent, avoid read data instructions in the second instruction queue that request data to be ready first (equivalent to read data instructions entering the slow path processing) from being continuously blocked due to their low priority and inability to compete for arbitration; it can also effectively guarantee the arbitration right of read data requests in the first sub-queue (equivalent to read data instructions entering the fast path processing), and when challenged, the requested data of the read data request can be returned, ensuring a fast response to read data instructions in the first instruction queue that request data to be ready first.

[0089] In some embodiments, the instruction submission module 122 is further configured to: select a target instruction queue from the first instruction queue and at least one candidate subqueue according to the first weight of the first instruction queue and the second weight of each candidate subqueue.

[0090] In this embodiment, the first weight of the first instruction queue and the second weight of each first sub-queue both indicate the probability of being selected as the target instruction queue. A larger first weight indicates a higher probability of the first instruction queue being selected as the target instruction queue; similarly, a larger second weight indicates a higher probability of the second instruction queue being selected as the target instruction queue. Furthermore, the second weights of each first sub-queue can be the same. Alternatively, at least some of the first sub-queues may have different second weights, which can be set according to actual application requirements. In this embodiment, the first weight of the first instruction queue is greater than the second weights of each candidate sub-queue, so that the instruction submission module 122 is more inclined to select the first instruction queue as the target instruction queue, ensuring that read data instructions requesting data in the first instruction queue that are ready first can be returned to the data requester more quickly.

[0091] The instruction submission module 122 can select the target instruction queue for the current clock cycle from the first instruction queue and each candidate sub-queue according to the target arbitration strategy, based on the first weight of the first instruction queue and the second weight of each candidate sub-queue. In some embodiments, the target arbitration strategy is a random strategy, and the first weight and the second weight are the arbitration probabilities of the first instruction queue and the first sub-queue. For example, if the first weight is 0.6 and the second weight is 0.4, the arbitration probability of the first instruction queue is greater than that of the first sub-queue. In other embodiments, the target arbitration strategy is a polling strategy, and the first weight and the second weight indicate the number of arbitrations for the first instruction queue and the first sub-queue in a single polling cycle. For example, if there are a first sub-queues, each with a second weight of 1, and the first instruction queue has a first weight of 6, then in a single polling cycle, the first instruction queue can be selected 6 times by the instruction submission module 122, and each first sub-queue can be selected once by the instruction submission module 122. Accordingly, it is easy to understand that the arbitration objects in a single polling cycle can be considered as a first sub-queues and 6 first instruction queues.

[0092] In the second alternative scenario, such as Figure 3 As shown, the second instruction queue includes: multiple first sub-queues and a second sub-queue.

[0093] The request merging module 11 is further configured to: when the current instruction state of a read data instruction is the second state, if the number of unread requests corresponding to the read data instruction is less than a target threshold, add the read data instruction to the second sub-queue; if the number of unread requests corresponding to the read data instruction is greater than or equal to the target threshold, determine the second thread bundle to which the thread sending the read data instruction belongs, and add the read data instruction to the first sub-queue corresponding to that second thread bundle. The number of unread requests corresponding to the read data instruction represents the number of read data requests in the read data instruction whose data readiness state is unready.

[0094] The response selection module 12 is further configured to: if a target read data instruction exists in the second sub-queue during the current clock cycle, migrate the target read data instruction to the first instruction queue, and select a target instruction queue from the first instruction queue and each of the first sub-queues. The current instruction state of the target read data instruction is the first state.

[0095] Here, the first sub-queue has a corresponding relationship with the thread bundle. For example, the correspondence between the first sub-queue and the thread bundle can be one-to-one, one-to-many, or many-to-one. That is, each sub-queue corresponds to one or more thread bundles. Each first sub-queue can be used to store at least one read data instruction processed by each thread in the thread bundle corresponding to the first sub-queue. In some embodiments, each first sub-queue is used to store instruction request information of read data instructions processed by each thread in the thread bundle corresponding to the first sub-queue. The request merging module 11 can add the instruction request information of the read data instruction to the first sub-queue corresponding to the thread bundle where the read data instruction is located when the current instruction state of the read data instruction is the second state. The instruction request information of the read data instruction may include, but is not limited to, at least one of the following: the first identification information of the read data instruction, the second identification information of at least one read data request corresponding to the read data instruction.

[0096] In some embodiments, after receiving at least one read data request corresponding to a read data instruction, the request merging module 11 can send each read data request of the read data instruction to the cache module 20, obtain the data readiness status of each read data request, and determine the current instruction status of the read data instruction based on the data readiness status of each read data request. This current instruction status is either a first status or a second status. Furthermore, if the current instruction status of the read data instruction is in the first status, the request merging module 11 adds the read data instruction to the first instruction queue. If the current instruction status of the read data instruction is in the second status, it counts the number of read data requests with a data readiness status of "not ready," obtains the "not ready request data" of the read data instruction, and determines whether the number of "not ready requests" of the read data instruction is less than a target threshold. Furthermore, if the number of "not ready requests" is less than the target threshold, the request merging module 11 adds the read data instruction to the second sub-queue; if the number of "not ready requests" corresponding to the read data instruction is greater than or equal to the target threshold, it determines the second thread bundle to which the thread sending the read data instruction belongs and adds the read data instruction to the first sub-queue corresponding to the second thread bundle.

[0097] In other embodiments, after sending each read data request to the cache module 20, the request merging module 11 can obtain the current instruction status of the read data instruction, which includes the data readiness status of each read data request corresponding to the read data instruction. The request merging module 11 determines whether the data readiness status of all read data requests is ready. If the data readiness status of all read data requests is ready, indicating that the current instruction status of the read data instruction is in the first state, the read data instruction is added to the first instruction queue. If the data readiness status of any read data request is not ready, indicating that the current instruction status of the read data instruction is in the second state, the number of read data requests with a data readiness status of not ready is counted to obtain the not ready request data of the read data instruction, and it is determined whether the number of not ready requests of the read data instruction is less than a target number threshold. Furthermore, if the number of unready requests is less than the target threshold, the request merging module 11 adds the read data instruction to the second sub-queue; if the number of unready requests corresponding to the read data instruction is greater than or equal to the target threshold, it determines the second thread bundle to which the thread sending the read data instruction belongs, and adds the read data instruction to the first sub-queue corresponding to that second thread bundle. The target threshold can be 1, 2, or y, where y is the product of the total number of read data requests and the target percentage, representing the percentage of unready requests in the total number of read data requests equal to the target percentage. For example, the target threshold is 1, or the target percentage is 1%.

[0098] The response selection module 12 can also migrate the target read data instruction to the first instruction queue and select the target instruction queue from the first instruction queue and each first sub-queue if the target read data instruction exists in the second sub-queue within a single clock cycle.

[0099] Here, the response selection module 12 can cyclically check the current instruction status of at least one read data instruction in the second sub-queue, and if a target read data instruction is found in the second sub-queue, the target read data instruction is moved to the first instruction queue. The specific implementation of the response selection module 12 checking the current instruction status of each read data instruction in the second sub-queue can be found in the aforementioned description of the response selection module 12 checking the current instruction status of each read data instruction in the second instruction queue; further comparison is not provided here.

[0100] In some embodiments, the request merging module 11 is further configured to: when the current instruction state of the read data instruction is the second state, if the number of unread requests corresponding to the read data instruction is less than the target number threshold and the read data instruction supports instruction-level out-of-order scheduling, then add the read data instruction to the second sub-queue; if the number of unread requests corresponding to the read data instruction is greater than or equal to the target number threshold, or if the read data instruction does not support instruction-level out-of-order scheduling, then determine the second thread bundle to which the thread sending the read data instruction belongs, and add the read data instruction to the first sub-queue corresponding to the second thread bundle.

[0101] The response selection module 12 is also used to: when there is a target read data instruction in the second sub-queue during the current clock cycle, migrate the target read data instruction to the first instruction queue, and select the target instruction queue from the first instruction queue and each first sub-queue, wherein the current instruction state of the target read data instruction is the first state.

[0102] In some embodiments, after receiving at least one read data request corresponding to a read data instruction, the request merging module 11 can send each read data request of the read data instruction to the cache module 20, obtain the data ready state of each read data request, and determine the current instruction state of the read data instruction based on the data ready state of each read data request. The current instruction state is either a first state or a second state.

[0103] Furthermore, when the current instruction state of the read data instruction is in the first state, the request merging module 11 determines whether the read data instruction supports instruction-level out-of-order scheduling. If the read data instruction supports instruction-level out-of-order scheduling, the read data instruction is added to the first instruction queue. If the read data instruction does not support instruction-level out-of-order scheduling, the second thread bundle to which the thread sending the read data instruction belongs is determined, and the read data instruction is added to the first sub-queue corresponding to the second thread bundle.

[0104] When the current instruction state of a read data instruction is in the second state, the request merging module 11 determines whether the read data instruction supports instruction-level out-of-order scheduling. If the read data instruction supports instruction-level out-of-order scheduling, it counts the number of read data requests in the read data instruction whose data ready state is not ready, obtains the unready request data for that read data instruction, and determines whether the number of unready requests for the read data instruction is less than a target threshold. If the number of unready requests is less than the target threshold, the read data instruction is added to the second sub-queue. If the number of unready requests corresponding to the read data instruction is greater than or equal to the target threshold, the second thread bundle to which the thread sending the read data instruction belongs is determined, and the read data instruction is added to the first sub-queue corresponding to that second thread bundle. Furthermore, if the read data instruction does not support instruction-level out-of-order scheduling, the request merging module 11 determines the second thread bundle to which the thread sending the read data instruction belongs and adds the read data instruction to the first sub-queue corresponding to that second thread bundle.

[0105] By temporarily storing read data instructions with a smaller number of unready requests in a second sub-queue, and having the response selection module 12 check the current instruction status of the read data instructions in the second sub-queue, read data instructions whose current instruction status has changed to the first state are migrated to the first instruction queue. This allows read data instructions with a smaller number of unready requests to be responded to more quickly than those with a larger number of unready requests. Since the smaller the number of unready requests for a read data instruction, the faster all read data requests corresponding to that instruction become ready, this scheme effectively ensures that read data instructions with ready requests are prioritized, allowing for faster returns to the data requester and guaranteeing the data processing performance of the memory merging device.

[0106] In some embodiments, please continue to refer to Figure 3 The response selection module 12 includes: The second data inspection module 123 is used to: cyclically check the current instruction status of at least one read data instruction in the second sub-queue, and if a target read data instruction is found in the second sub-queue, migrate the target read data instruction to the first instruction queue.

[0107] Multiple first data checking modules 121 are respectively used to check the current instruction status of the read data instruction at the head of their respective first sub-queues within the current clock cycle.

[0108] The instruction submission module 122 is used to: select a target instruction queue from the first instruction queue and each of the first sub-queues within the current clock cycle.

[0109] Here, the first data checking module 121 and the instruction submission module 122 are the same as the first data checking module 121 and the instruction submission module 122 involved in the response selection module 12 in the first case mentioned above, and will not be described in detail here.

[0110] The second data checking module 123 can cyclically check the current instruction status of at least one read data instruction in the second sub-queue, and if it detects a target read data instruction in the second sub-queue with a current instruction status of the first state, it will move the target read data instruction to the first instruction queue. It should be noted that the implementation method of the second data checking module 123 checking the current instruction status of read data instructions in the second sub-queue can be the same as the implementation method of the first data checking module 121 checking the current instruction status of the read data instruction at the head of its corresponding first sub-queue; a comparison will not be elaborated upon. The main difference is that the second data checking module 123 can check one or more read data instructions in the second sub-queue in a single round; the first data checking module 121 only checks the read data instruction at the head of its corresponding first sub-queue.

[0111] In some embodiments, the second data checking module 123 can cyclically check the current instruction status of at least one read data instruction in the second sub-queue, and after a single round of checking, if a target read data instruction exists in the second sub-queue, the target read data instruction is moved to the first instruction queue. In other embodiments, the second data checking module 123 can also move the target read data instruction to the first instruction queue if a target read data instruction is detected during the checking process. In some embodiments, the second sub-queue is a FIFO queue. The second data checking module 123 can cyclically check the current instruction status of at least one read data instruction in the second sub-queue, and after a single round of checking, if the read data instruction at the head of the second sub-queue is a target read data instruction, the read data instruction at the head of the queue is moved to the first instruction queue, until the read data instruction at the head of the second sub-queue is no longer a target read data instruction.

[0112] For example, the second sub-queue is a FIFO queue. The second data checking module 123 can cyclically check the current instruction status of all read data instructions in the second sub-queue, and when a target read data instruction is found in the second sub-queue, the target read data instruction is moved to the first instruction queue. It is easy to understand that in this embodiment, the single-round check object of the second data checking module 123 is all read data instructions in the second sub-queue. Specifically, for example, the second sub-queue includes read data instruction 1, read data instruction 2, read data instruction 3, and read data instruction 4. The second data checking module 123 cyclically checks the current instruction status of read data instruction 1, read data instruction 2, read data instruction 3, and read data instruction 4 in sequence, so that when the current instruction status of read data instruction 1 and 3 changes to the first state, read data instruction 1 is moved to the first instruction queue. Then, the second data checking module 123 checks the current instruction status of read data instruction 2, read data instruction 3 and read data instruction 4 in a loop. When the current instruction status of read data instruction 2 and 3 changes to the first state, read data instruction 2 and 3 are moved to the first instruction queue in sequence, and so on.

[0113] In some embodiments, the second data checking module 123 can cyclically check the data readiness status of each read data request of at least one read data instruction in the second sub-queue in the cache module 20 to obtain the current instruction status of the read data instruction. That is, for a single read data instruction, the second data checking module 123 obtains the current instruction status of the read data instruction by checking the data readiness status of all read data requests of the read data instruction.

[0114] Alternatively, the second data checking module 123 can cyclically check the data readiness status of at least one target read data request in the second sub-queue within the cache module 20 to obtain the current instruction status of that read data instruction. The target read data request is the read data request whose data readiness status was not ready in the previous cyclic check. That is, for a single read data instruction, the second data checking module 123 obtains the current instruction status of the read data instruction by only checking the data readiness status of the target read data request of that read data instruction. Specifically, the second data checking module 123 can determine the current instruction status of the read data instruction as the first state when the data readiness status of each target read data request changes from not ready to ready; and determine the current instruction status of the read data instruction as the second state when the data readiness status of any target read data request remains not ready.

[0115] Clearly, since the number of unread requests for each read data instruction in the second sub-queue is relatively small, meaning there are fewer read data instructions in the second sub-queue with a data readiness status of unread, the second data checking module 123 determines the current instruction status of a single read data instruction by checking only the data readiness status of read data requests with an unread data readiness status. Compared to checking the data readiness status of all read data requests for a single read data instruction, the former requires significantly fewer clock cycles per round of checking, effectively improving the processing efficiency of the second data checking module 123.

[0116] In other embodiments, the second data checking module 123 is used for one of the following: Loop through and check the data readiness status of the target read data requests corresponding to the first n read data instructions in the second sub-queue in the cache module. The system iteratively checks the read data status of the m target read data requests corresponding to the read data instructions in the second sub-queue in the cache module. The target read data requests are those whose read data status was not ready in the previous iterative check.

[0117] The second data checking module 123 is used to cyclically check the data readiness status of the target read data requests corresponding to the first n read data instructions in the second sub-queue in the cache module, so as to obtain the current instruction status of at least one read data instruction in the second sub-queue. Here, the single-round check object of the second data checking module 123 is: the target read data requests corresponding to the first n read data instructions in the second sub-queue. For example, the second sub-queue is a FIFO queue. The second sub-queue includes, in sequence: read data instruction 1 at the head of the queue, read data instruction 2, read data instruction 3, and read data instruction 4. Assume that n is 2. The second data checking module 123 can cyclically check the current instruction status of read data instruction 1 and read data instruction 2 in sequence, so that when the current instruction status of read data instruction 1 changes to the first state, read data instruction 1 is moved to the first instruction queue. Then, the second data checking module 123 sequentially checks the read data instruction 2 and read data instruction 3, so that when the current instruction state of read data instruction 2 changes to the first state, read data instruction 2 is moved to the first instruction queue. Then, the second data checking module 123 sequentially checks the read data instruction 3 and read data instruction 4, and so on.

[0118] The second data checking module 123 is used to cyclically check the data readiness status of m read data requests corresponding to read data instructions in the second sub-queue in the cache module, so as to obtain the current instruction status of at least one read data instruction in the second sub-queue. Here, the single-round check object of the second data checking module 123 is: the m read data requests corresponding to read data instructions in the second sub-queue. It should be noted that the m read data requests can also be the m read data requests of any read data instruction in the second sub-queue. Alternatively, the m read data requests can be the m read data requests of the preceding read data instructions in the second sub-queue. For example, the second sub-queue includes, in sequence, the read data instruction instruction 1, read data instruction instruction 2, and read data instruction instruction 3 at the head of the queue. The number of target read data requests corresponding to read data instruction instruction 1, read data instruction instruction 2, and read data instruction instruction 3 are 2, 3, and 1, respectively. If m is 3, then the m target read data requests can refer to: the two target read data requests corresponding to read data instruction 1, and the one target read data request corresponding to read data instruction 2. Clearly, compared to the second data inspection module 123's method of checking the data readiness status of any m target read data requests in the second sub-queue at a time, the second data inspection module 123's method of checking the data readiness status of the m read data requests of the preceding read data instruction at a time allows the data readiness status check to be more focused on a specific read data instruction. This enables a faster determination of the data readiness status of all read data requests for a specific read data instruction, improving the efficiency of the second data inspection module 123 in checking the current instruction status of read data instructions.

[0119] Here, n and m are both positive integers, and their values ​​can be statically configured according to actual needs or dynamically configured using registers. The smaller the values ​​of n and m, the fewer clock cycles required for a single round of checking by the second data checking module 123. Clearly, whether the objects checked in a single round by the second data checking module 123 are the target read data requests corresponding to the first n read data instructions in the second sub-queue, or the m read data requests corresponding to the read data instructions in the second sub-queue, the number of clock cycles required for a single round of checking by the second data checking module 123 can be effectively reduced by setting the values ​​of n and m, thereby improving the processing efficiency of the second data checking module 123. In particular, when the second sub-queue is a FIFO queue, since the second sub-queue needs to follow the first-in-first-out principle, and the migration operation of the read data instruction at the head of the second sub-queue when it becomes a target read data instruction needs to be executed after the second data checking module 123 has completed checking all objects in a single round, the number of clock cycles required for a single round of checking by the second data checking module 123 will affect the migration efficiency of the read data instruction at the head of the second sub-queue. The more objects the second data inspection module 123 inspects in a single round, the longer the waiting time required for the migration of read data instructions at the head of the second sub-queue. Therefore, by setting the values ​​of n and m, the waiting time required for the migration of read data instructions at the head of the second sub-queue can be effectively controlled, thereby improving the migration efficiency of read data instructions at the head of the second sub-queue and increasing the data processing efficiency of the second data inspection module 123.

[0120] In some embodiments, each element in the first instruction queue and each element in the second sub-queue includes a read data instruction and a jump pointer. The jump pointer is a pointer to the storage space where the next element of the first element is located.

[0121] The second data checking module 123 is further configured to: exchange storage information between the storage space where the target read data instruction is located and the storage space where the element to be adjusted is located; update the jump pointer of the tail element in the first instruction queue to the pointer of the storage space where the target read data instruction is located; and update the jump pointer of the element where the target read data instruction is located to a null value. The element to be adjusted is the element following the element where the target read data instruction is located.

[0122] Here, the first instruction queue and the second sub-queue can be linked list queues. Elements in the first instruction queue (which can also be understood as storage entries) and elements in the second instruction queue can be linked through jump pointers. The second data inspection module 123 migrates the target read data instruction from the second sub-queue to the first instruction queue. This can be understood as breaking the chain, retrieving the element containing the target read data instruction from the second sub-queue, and re-linking the remaining elements in the second sub-queue; then, linking this element to the tail of the first instruction queue.

[0123] Specifically, the second data checking module 123 can exchange storage information between the storage space where the target read data instruction is located and the storage space where the element to be adjusted is located. This allows the element containing the target read data instruction to be transferred to the first storage space, and the element to be adjusted to be transferred to the second storage space. Based on the jump pointer in the element preceding the element containing the target read data instruction in the second sub-queue, the element to be adjusted can be read. The first storage space is the storage space where the element to be adjusted was located before the transfer, and the second storage space is the storage space where the target read data instruction was located before the transfer. Furthermore, the second data checking module 123 updates the jump pointer of the tail element in the first instruction queue to the pointer of the storage space where the target read data instruction is located, so that the jump pointer of the tail element in the first instruction queue can point to the first storage space, facilitating the reading of the element containing the target read data instruction based on the jump pointer of the tail element. Additionally, the second data checking module 123 updates the jump pointer of the element containing the target read data instruction to a null value to prevent reading data in the first storage space via the jump pointer of the element containing the target read data instruction, thus realizing the migration of the target read data instruction from the second sub-queue to the first instruction queue.

[0124] In some embodiments, the elements in the first instruction queue and the elements in the second sub-queue further include: an element identifier. The second data inspection module 123 is further configured to: exchange the storage information other than the element identifier in the storage space where the target read data instruction is located with the storage information other than the element identifier in the storage space where the element to be adjusted is located.

[0125] Here, the elements in the first instruction queue and the elements in the second sub-queue both include: the element identifier instruction entry id, the data read instruction instruction, and the jump pointer next instruction ptr.

[0126] In some embodiments, the second data checking module 123 can, when a target read data instruction exists in the second sub-queue, read the first storage information of the second storage space where the target read data instruction is located in the second sub-queue, and read the second storage information of the first storage space where the element to be adjusted is located in the second sub-queue. Then, it writes the storage information in the second storage information, excluding the element identifier, into the second storage space, updates the jump pointer in the first storage information to a null value, and writes the storage information in the first storage information, excluding the element identifier, into the first storage space. Afterwards, the second data checking module 123 reads the tail element in the first instruction queue, updates the jump pointer of the tail element to the pointer in the first storage space, and re-stores it, thereby realizing the migration of the target read data instruction from the second sub-queue to the first instruction queue.

[0127] In some embodiments, the elements in the first instruction queue and the elements in the second instruction queue may also synchronously include: read data instructions, and cache request info corresponding to each read data instruction. The request merging module 11 may synchronously write each read data request corresponding to the read data instruction when writing the read data instruction to the first instruction queue or the second instruction queue (first sub-queue or second sub-queue).

[0128] For example, such as Figure 4 and Figure 5 As shown, assume the second sub-queue has four elements: element 1, element 2, element 3, and element 4. Element 1 includes: element identifier instruction entry id 0, read data instruction instruction1_1 and its corresponding read data request cache request info, and jump pointer next instruction ptr 1_3; element 2 includes: element identifier instruction entry id 3, read data instruction instruction1_2 and its corresponding read data request cache request info, and jump pointer next instruction ptr 1_5; element 3 includes: element identifier instruction entry id 5, read data instruction instruction1_3 and its corresponding read data request cache request info, and jump pointer next instruction ptr 1_7; element 4 includes: element identifier instruction entry id 7, read data instruction instruction1_4 and its corresponding read data request cache request info, and null value.

[0129] The first instruction queue has two elements, element 1 and element 2. Element 1 includes: an element identifier (instructionentry id 100), a read data instruction (instruction 2_100) and its corresponding read data request (cache request info), and a jump pointer (next instruction ptr 2_101). Element 2 includes: an element identifier (instruction entryid 101), a read data instruction (instruction2_101) and its corresponding read data request (cache request info), and a null value.

[0130] Assuming the target read data instruction is read data instruction instruction2, the migration process of the target read data instruction from the second sub-queue to the first instruction queue is broken down into two parts: the unlinking and relinking of the second sub-queue, and the relinking of the first instruction queue.

[0131] The process by which the second data inspection module 123 dischains the element containing the target read data instruction in the second sub-queue and rechains the remaining elements in the second sub-queue can be detailed in the following example: Figure 4 .from Figure 4 It can be seen that the second data inspection module 123 moves the read data instruction 3 and its corresponding read data request cache requestinfo, jump pointer next instruction ptr 7 in element 3 to the storage space where element 2 is located. Thus, the link relationship of the second sub-queue is updated from element identifier instruction entry id 0 ---> element identifier instruction entry id 3 ---> element identifier instruction entry id 5 ---> element identifier instruction entry id 7 to: element identifier instruction entry id 0 ---> element identifier instruction entry id 3 ---> element identifier instruction entry id 7.

[0132] The process by which the second data inspection module 123 links the element containing the target read data instruction to the tail of the first instruction queue can be detailed in the following example. Figure 5 .from Figure 5It can be seen that the second data inspection module 123 can move the read data instruction instruction1_2 and its corresponding read data request cache request info, jump pointer nextinstruction ptr 1_5 in element 2 of the second sub-queue to the storage space where element 3 is located, and update the jump pointer in the element where the element identifier instruction entryid 101 is located from null to the jump pointer next instruction ptr 1_5, and update the jump pointer stored in the storage space where element 3 is located (i.e. the jump pointer in the element where the element identifier instruction entryid 5 is located) from the jump pointer next instruction ptr 1_5 to null. In this way, the link relationship of the first instruction queue is updated from element identifier instruction entry id 100 ---> element identifier instruction entry id 101 to: element identifier instruction entry id 100 ---> element identifier instruction entry id 101 ---> element identifier instruction entry id 5, effectively realizing the migration of target read data instructions from the second sub-queue to the first instruction queue.

[0133] It should be noted that in some embodiments, the first instruction queue and the second sub-queue may also be a value queue, or a stack queue or other forms of queue, and this application does not limit them.

[0134] In this embodiment, the memory merging device includes a request merging module, a response selection module, and a data return module. The request merging module, based on a received read data instruction, sends at least one read data request corresponding to the read data instruction to the cache module, obtaining the current instruction state of the read data instruction. If the current instruction state is a first state, the read data instruction is added to a first instruction queue; if the current instruction state is a second state, the read data instruction is added to a second instruction queue. The response selection module, within the current clock cycle, checks the current instruction state of the read data instruction at the head of the second instruction queue and selects a target instruction queue from the first and second instruction queues, where the current instruction state of the read data instruction at the head of the target instruction queue is the first state. The data return module retrieves the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and returns it to the requester of the first read data instruction. The first state indicates that the data ready state of each read data request corresponding to the read data instruction is ready, and the second state indicates that the data ready state of any read data request corresponding to the read data instruction is not ready. The read-ready condition indicates that the requested data has been hit in the cache module and can be read by the memory merging device.

[0135] In this way, the request merging module categorizes read data instructions into a first instruction queue and a second instruction queue based on whether all read data requests corresponding to each read data instruction are ready. This allows the response selection module to choose the target instruction queue from the two queues, ensuring that the first instruction queue, with all read data requests already ready, is more likely to be selected than the second instruction queue. Consequently, the data return module reads and returns the read data request from the first instruction queue, effectively guaranteeing that read data requests with ready data are returned to the requester more quickly. Furthermore, since the second instruction queue may also contain earlier-written read data requests that are ready, it also has a chance of being selected as the target instruction queue by the response selection module and processed by the data return module to return the requested data. Therefore, the technical solution disclosed herein can not only achieve a fast response to read data instructions that request data to be ready first, but also effectively ensure that other read data instructions can be responded to when the requested data is ready, reducing the probability of read data instruction blocking and ensuring the stability of data access performance of the memory merging device under the fast response mechanism.

[0136] This disclosure provides a data processing method applied to a memory merging apparatus, which includes a request merging module, a response selection module, and a data return module. Figure 6As shown, the data processing method includes the following steps S601 to S603: Step S601: The request merging module sends at least one read data request corresponding to the received read data instruction to the cache module to obtain the current instruction status of the read data instruction; if the current instruction status is the first status, the read data instruction is added to the first instruction queue; if the current instruction status is the second status, the read data instruction is added to the second instruction queue; the first status indicates that the data ready status of each read data request corresponding to the read data instruction is ready, and the second status indicates that the data ready status of any read data request corresponding to the read data instruction is not ready.

[0137] In step S602, the response selection module checks the current instruction status of the read data instruction at the head of the second instruction queue within the current clock cycle, and selects the target instruction queue from the first instruction queue and the second instruction queue. The current instruction status of the first read data instruction at the head of the target instruction queue is the first status. In step S603, the data return module retrieves the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and returns it to the requester of the first read data instruction.

[0138] In this embodiment, the request merging module categorizes read data instructions into a first instruction queue and a second instruction queue based on whether all read data requests corresponding to each read data instruction are ready. This allows the response selection module to select the target instruction queue from the two queues, ensuring that the first instruction queue, with all read data requests already ready, is more likely to be selected than the second instruction queue. Consequently, the data return module reads and returns the request data from the first instruction queue, effectively prioritizing read data instructions with ready data for faster return to the requester. Furthermore, since the second instruction queue may also have all read data requests already ready when the response selection module selects the target instruction queue, it also has a certain probability of being selected as the target instruction queue and processed by the data return module to return the requested data. Therefore, the technical solution disclosed herein can not only achieve a fast response to read data instructions that request data to be ready first, but also effectively ensure that other read data instructions can be responded to when the requested data is ready, reducing the probability of read data instruction blocking and ensuring the stability of data access performance of the memory merging device under the fast response mechanism.

[0139] In some embodiments, the second instruction queue includes: a plurality of first sub-queues, each first sub-queue corresponding to one or more thread bundles; step S601, in which the read data instruction is added to the second instruction queue when the current instruction state is the second state, may include step S611: In step S611, if the current instruction state of the read data instruction is the second state, the request merging module determines the thread bundle to which the thread that sent the read data instruction belongs, and adds the read data instruction to the first sub-queue corresponding to the thread bundle.

[0140] In some embodiments, the response selection module includes: a plurality of first data checking modules and an instruction submission module, wherein the first data checking modules have a one-to-one relationship or a one-to-many relationship with the first sub-queue; step S602, the response selection module checks the current instruction status of the read data instruction at the head of the second instruction queue in the current clock cycle, and selects the target instruction queue from the first instruction queue and the second instruction queue, which may include steps S621 to S622.

[0141] Step S621: The first data checking module checks the current instruction status of the read data instruction at the head of the corresponding first sub-queue within the current clock cycle. In step S622, the instruction submission module selects a target instruction queue from the first instruction queue and at least one candidate sub-queue within the current clock cycle. The at least one candidate sub-queue is the sub-queue whose current instruction state is the first state of the read data instruction at the head of the multiple first sub-queues.

[0142] In some embodiments, step S622, where the instruction submission module selects a target instruction queue from at least one candidate sub-queue among the first instruction queue and the first sub-queue within the current clock cycle, may include step S631: Step S631: The instruction submission module selects a target instruction queue from the first instruction queue and at least one candidate sub-queue according to the first weight of the first instruction queue and the second weight of each candidate sub-queue, wherein the first weight is greater than the second weight.

[0143] In some embodiments, the second instruction queue includes: a plurality of first sub-queues and a second sub-queue, each first sub-queue corresponding to one or more thread bundles; Step S601, when the current instruction state is the second state, adds the read data instruction to the second instruction queue, which may include step S641: In step S641, if the current instruction state of the read data instruction is the second state, and the number of unread requests corresponding to the read data instruction is less than the target number threshold, the request merging module adds the read data instruction to the second sub-queue. If the number of unread requests corresponding to the read data instruction is greater than or equal to the target number threshold, the thread bundle to which the thread sending the read data instruction belongs is determined, and the read data instruction is added to the first sub-queue corresponding to that thread bundle. The number of unread requests corresponding to the read data instruction represents the number of read data requests in the read data instruction whose data read state is not ready. Step S602: The response selection module checks the current instruction status of the read data instruction at the head of the second instruction queue within the current clock cycle, and selects the target instruction queue from the first and second instruction queues. This may include step S642: In step S642, if the target read data instruction exists in the second sub-queue during the current clock cycle, the response selection module migrates the target read data instruction to the first instruction queue and selects the target instruction queue from the first instruction queue and each first sub-queue. The current instruction state of the target read data instruction is the first state.

[0144] In some embodiments, the response selection module includes: a second data inspection module, a plurality of first data inspection modules, and an instruction submission module; step S642: if the response selection module has a target read data instruction in the second sub-queue during the current clock cycle, it migrates the target read data instruction to the first instruction queue, and selects a target instruction queue from the first instruction queue and each of the first sub-queues, wherein the current instruction state of the target read data instruction is the first state, including steps S651 to S653: Step S651: The second data checking module cyclically checks the current instruction status of at least one read data instruction in the second sub-queue, and if a target read data instruction is found in the second sub-queue, the target read data instruction is moved to the first instruction queue. Step S652: Multiple first data checking modules check the current instruction status of the read data instruction at the head of their respective first sub-queues within the current clock cycle. In step S653, the instruction submission module selects the target instruction queue from the first instruction queue and each of the first sub-queues within the current clock cycle.

[0145] In some embodiments, the second data checking module in step S651 cyclically checks the current instruction status of at least one read data instruction in the second sub-queue, which may include one of steps S661 and S662: Step S661: Loop through and check the data readiness status of the target read data requests corresponding to the first n read data instructions in the second sub-queue in the cache module; Step S662: Iteratively check the data readiness status of the m target read data requests corresponding to the read data instructions in the second sub-queue in the cache module. The target read data requests are the read data requests whose data readiness status was not ready in the previous iterative check.

[0146] In some embodiments, the elements in the first instruction queue and the elements in the second sub-queue both include: a read data instruction and a jump pointer, wherein the jump pointer is a pointer to the storage space where the next element of the element is located; the migration of the target read data instruction to the first instruction queue in step S651 may include step S671: In step S671, the second data inspection module exchanges the storage information in the storage space where the target read data instruction is located and the storage space where the element to be adjusted is located, updates the jump pointer of the tail element in the first instruction queue to the pointer of the storage space where the target read data instruction is located, and updates the jump pointer of the element where the target read data instruction is located to null. The element to be adjusted is the next element after the element where the target read data instruction is located.

[0147] In some embodiments, the elements in the first instruction queue and the elements in the second sub-queue further include: an element identifier; the second data inspection module exchanging the storage information in the storage space where the target read data instruction is located and the storage space where the element to be adjusted is located in step S671 may include step S681: In step S681, the second data inspection module exchanges the storage information (excluding the element identifier) ​​in the storage space where the target read data instruction is located with the storage information (excluding the element identifier) ​​in the storage space where the element to be adjusted is located.

[0148] In some embodiments, when the current instruction state is the first state, adding the read data instruction to the first instruction queue in step S601 may include step S691: Step S691: If the current instruction state of the read data instruction is in the first state and the read data instruction supports instruction-level out-of-order scheduling, the request merging module adds the read data instruction to the first instruction queue.

[0149] In this embodiment, the request merging module categorizes read data instructions into a first instruction queue and a second instruction queue based on whether all data requested by each read data instruction is ready. This allows the response selection module to select the target instruction queue from the two queues, ensuring that the first instruction queue, with all data requested by its read data instructions already ready, is more likely to be selected than the second instruction queue. Consequently, the data return module reads and returns the data requested by the first read data instruction in the first instruction queue, effectively guaranteeing that read data instructions with ready data are returned to the requester more quickly. Furthermore, since the data requested by earlier read data instructions in the second instruction queue may also be ready, the second instruction queue may also be selected as the target instruction queue by the response selection module and processed by the data return module to return the requested data. Therefore, the technical solution disclosed herein can not only achieve a fast response to read data instructions that request data to be ready first, but also effectively ensure that other read data instructions can be responded to when the requested data is ready, reducing the probability of read data instruction blocking and ensuring the stability of data access performance of the memory merging device under the fast response mechanism.

[0150] The descriptions of the above method embodiments are similar to those of the above device embodiments, and have similar beneficial effects. In some embodiments, the functions or modules included in the device provided in this disclosure can be used to perform the methods described in the above method embodiments. For technical details not disclosed in the method embodiments of this disclosure, please refer to the descriptions of the device embodiments of this disclosure for understanding.

[0151] It should be noted that the implementation of the embodiments disclosed herein is not limited to any specific hardware, software or firmware, or any combination of hardware, software and firmware.

[0152] This disclosure provides a chip product, such as... Figure 7 As shown, the chip product 70 includes the memory merging device 10 in the above embodiments. For example, the chip product may include, but is not limited to, at least one of a GPU chip, an AI chip, a TPU (Tensor Processing Unit) chip, an NPU (Neural Processing Unit) chip, etc.

[0153] This disclosure provides a computer device, such as... Figure 8As shown, the computer device 80 includes the memory merging device 10 in the above embodiments. The computer device 80 may include, but is not limited to, personal computers, workstations, game consoles, or some mobile devices (such as tablets, smartphones, etc.), and may also be in-vehicle terminal devices, smart home devices, smart TVs, or smart robots, etc. It may also be servers, server clusters, artificial intelligence computing clusters, or cloud computing clusters, etc., wherein the artificial intelligence computing cluster may also be simply referred to as an intelligent computing cluster or smart computing cluster, and the embodiments disclosed herein do not limit this.

[0154] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above method, chip product, and device embodiments are similar to the descriptions of the above apparatus embodiments and have similar beneficial effects. For technical details not disclosed in the method, chip product, and device embodiments of this disclosure, please refer to the descriptions of the method embodiments of this disclosure for understanding.

[0155] It should be understood that the phrase "an embodiment" or "one embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this disclosure. Therefore, "in one embodiment" or "one embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this disclosure, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure. The sequence numbers of the above embodiments of this disclosure are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0156] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0157] In the several embodiments provided in this disclosure, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components may be combined, or integrated into another system, or some features may be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed may be through some interfaces, and the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0158] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units; some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, the functional units in the embodiments of this disclosure may all be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the integrated unit may be implemented in hardware or in a combination of hardware and software functional units.

[0159] The above are merely exemplary embodiments of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A memory merging device, characterized in that, include: The request merging module is used to: send at least one read data request corresponding to the received read data instruction to the cache module based on the received read data instruction, and obtain the current instruction status of the read data instruction; When the current instruction state is in the first state, the read data instruction is added to the first instruction queue; when the current instruction state is in the second state, the read data instruction is added to the second instruction queue. The first state indicates that the data ready state of each read data request corresponding to the read data instruction is ready, and the second state indicates that the data ready state of any read data request corresponding to the read data instruction is not ready. The response selection module is used to: check the current instruction status of the read data instruction at the head of the second instruction queue during the current clock cycle, and select a target instruction queue from the first instruction queue and the second instruction queue, wherein the current instruction status of the first read data instruction at the head of the target instruction queue is the first status; The data return module is used to: retrieve the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and return it to the requester of the first read data instruction.

2. The memory merging apparatus according to claim 1, characterized in that, The second instruction queue includes: a plurality of first sub-queues, each of which corresponds to one or more thread bundles; The request merging module is further configured to: when the current instruction state of the read data instruction is the second state, determine the first thread bundle to which the thread that sent the read data instruction belongs, and add the read data instruction to the first sub-queue corresponding to the first thread bundle.

3. The memory merging apparatus according to claim 2, characterized in that, The response selection module includes: Multiple first data checking modules, each first data checking module having a one-to-one or one-to-many relationship with the first sub-queue, the first data checking module being used to: check the current instruction status of the read data instruction at the head of the corresponding first sub-queue within the current clock cycle; The instruction submission module is configured to: select a target instruction queue from the first instruction queue and at least one candidate sub-queue during the current clock cycle, wherein the at least one candidate sub-queue is a sub-queue whose current instruction state of the read data instruction at the head of the plurality of first sub-queues is the first state.

4. The memory merging apparatus according to claim 3, characterized in that, The instruction submission module is also used for: A target instruction queue is selected from the first instruction queue and at least one candidate sub-queue based on the first weight of the first instruction queue and the second weight of each candidate sub-queue, wherein the first weight is greater than the second weight.

5. The memory merging apparatus according to claim 1, characterized in that, The second instruction queue includes: multiple first sub-queues and a second sub-queue, each of the first sub-queues corresponding to one or more thread bundles; The request merging module is further configured to: when the current instruction state of the read data instruction is the second state, if the number of unread requests corresponding to the read data instruction is less than a target number threshold, add the read data instruction to the second sub-queue; if the number of unread requests corresponding to the read data instruction is greater than or equal to the target number threshold, determine the second thread bundle to which the thread that sent the read data instruction belongs, and add the read data instruction to the first sub-queue corresponding to the second thread bundle, wherein the number of unread requests corresponding to the read data instruction represents the number of read data requests in the read data instruction whose data read state is unread. The response selection module is further configured to: when there is a target read data instruction in the second sub-queue during the current clock cycle, migrate the target read data instruction to the first instruction queue, and select a target instruction queue from the first instruction queue and each of the first sub-queues, wherein the current instruction state of the target read data instruction is the first state.

6. The memory merging apparatus according to claim 5, characterized in that, The response selection module includes: The second data inspection module is used to: cyclically check the current instruction status of at least one read data instruction in the second sub-queue, and if the target read data instruction is found in the second sub-queue, migrate the target read data instruction to the first instruction queue; Multiple first data checking modules are respectively used to: check the current instruction status of the read data instruction of the head of the corresponding first sub-queue within the current clock cycle; The instruction submission module is used to: select a target instruction queue from the first instruction queue and each of the first sub-queues within the current clock cycle.

7. The memory merging apparatus according to claim 6, characterized in that, The second data inspection module is used for one of the following: The system iteratively checks the data readiness status of the target read data requests corresponding to the first n read data instructions in the second sub-queue within the cache module. The system iteratively checks the data readiness status of m target read data requests corresponding to read data instructions in the second sub-queue within the cache module. The target read data requests are those whose data readiness status was not ready in the previous iterative check.

8. The memory merging apparatus according to claim 6, characterized in that, The elements in the first instruction queue and the elements in the second sub-queue each include: a read data instruction and a jump pointer, wherein the jump pointer is a pointer to the storage space where the next element of the given element is located; The second data inspection module is further configured to: exchange storage information in the storage space where the target read data instruction is located and the storage space where the element to be adjusted is located, update the jump pointer of the tail element in the first instruction queue to the pointer of the storage space where the target read data instruction is located, and update the jump pointer of the element where the target read data instruction is located to a null value, wherein the element to be adjusted is the next element after the element where the target read data instruction is located.

9. The memory merging apparatus according to claim 8, characterized in that, The elements in the first instruction queue and the elements in the second sub-queue each include: an element identifier; the second data checking module is further configured to: exchange the storage information other than the element identifier in the storage space where the target read data instruction is located with the storage information other than the element identifier in the storage space where the element to be adjusted is located.

10. The memory merging apparatus according to claim 1, characterized in that, The request merging module is also used for: If the current instruction state of the read data instruction is the first state and the read data instruction supports instruction-level out-of-order scheduling, the read data instruction is added to the first instruction queue.

11. The memory merging apparatus according to claim 10, characterized in that, The second instruction queue includes: multiple first sub-queues and a second sub-queue, each of the first sub-queues corresponding to one or more thread bundles; The request merging module is further configured to: when the current instruction state of the read data instruction is the second state, if the number of unread requests corresponding to the read data instruction is less than the target number threshold and the read data instruction supports instruction-level out-of-order scheduling, then add the read data instruction to the second sub-queue; if the number of unread requests corresponding to the read data instruction is greater than or equal to the target number threshold, or if the read data instruction does not support instruction-level out-of-order scheduling, then determine the second thread bundle to which the thread sending the read data instruction belongs, and add the read data instruction to the first sub-queue corresponding to the second thread bundle, wherein the number of unread requests corresponding to the read data instruction represents the number of read data requests in the read data instruction whose data read state is unread; The response selection module is further configured to: when there is a target read data instruction in the second sub-queue during the current clock cycle, migrate the target read data instruction to the first instruction queue, and select a target instruction queue from the first instruction queue and each of the first sub-queues, wherein the current instruction state of the target read data instruction is the first state.

12. A data processing method, characterized in that, The method is applied to a memory merging apparatus, which includes a request merging module, a response selection module, and a data return module; the method includes: The request merging module sends at least one read data request corresponding to the received read data instruction to the cache module to obtain the current instruction status of the read data instruction; if the current instruction status is a first status, the read data instruction is added to a first instruction queue; if the current instruction status is a second status, the read data instruction is added to a second instruction queue; the first status indicates that the data ready status of each read data request corresponding to the read data instruction is ready, and the second status indicates that the data ready status of any read data request corresponding to the read data instruction is not ready. Within the current clock cycle, the response selection module checks the current instruction status of the read data instruction at the head of the second instruction queue and selects a target instruction queue from the first instruction queue and the second instruction queue. The current instruction status of the first read data instruction at the head of the target instruction queue is the first status. The data return module retrieves the target data of each read data request in the first read data instruction at the head of the target instruction queue from the cache module and returns it to the requester of the first read data instruction.

13. A chip product, characterized in that, The chip product includes the memory merging device as described in any one of claims 1 to 11.

14. A computer device, characterized in that, The computer device includes a memory merging apparatus as described in any one of claims 1 to 11.

Images