Data processing system, method, device, medium and program product

Abstract

The invention discloses a data processing system, method and equipment, a medium and a program product in the technical field of computers. In the invention, a cache coherence root port of a computing device is connected with a cache coherence sub-port of a memory processing device through a cache coherence protocol, so that the computing device is used for synchronizing a query matrix of a model processing task and a first result output in a key value matrix calculation stage to the memory processing device through the cache coherence protocol; meanwhile, the memory processing device can synchronize the second result to the computing device through the cache coherence protocol, so that rapid data transmission between the computing device and the memory processing device is realized, the memory space of the computing device is saved, the overhead of a host processor in the execution process of the model processing task is reduced, and the performance of the model processing task is improved. And the task processing efficiency of the hardware computing equipment can be improved to a certain extent.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data processing system, method, device, medium and program product. Background Art

[0002] Currently, when using hardware devices to perform tasks such as model inference, due to the limited memory space in the hardware devices, task-related data can be stored in the host. However, this requires the hardware devices to frequently interact with the host during the execution of model inference tasks, which increases the overhead of the host processor and reduces the task processing efficiency of the hardware devices.

[0003] Therefore, how to improve the efficiency of hardware devices in performing tasks is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, an object of the present invention is to provide a data processing system, method, device, medium and program product to improve the efficiency of hardware devices in performing tasks.

[0005] In a first aspect, the present invention provides a data processing system, comprising: a host, a computing device and a memory processing device; the host is connected to the computing device and the memory processing device; the cache consistency root port of the computing device is connected to the cache consistency subport of the memory processing device through a cache consistency protocol; the host is used to: send a model processing task to the computing device; the computing device is used to: execute the pre-filling phase and the query matrix and key-value matrix calculation phase of the model processing task, and synchronize the first result output by the query matrix and key-value matrix calculation phase to the memory processing device through the cache consistency protocol; the memory processing device is used to: execute the attention calculation phase of the model processing task based on the first result, and synchronize the second result output by the attention calculation phase to the computing device through the cache consistency protocol; the computing device is used to: determine the task result of the model processing task based on the second result.

[0006] In a second aspect, the present invention provides a data processing method, which is applied to a computing device in a data processing system, including: executing a pre-filling stage and a query matrix and key value matrix calculation stage of a model processing task sent by a host in the data processing system; synchronizing a first result output from the query matrix and key value matrix calculation stage to a memory processing device in the data processing system through a cache consistency protocol, so that the memory processing device executes the attention calculation stage of the model processing task based on the first result, and synchronizes the second result output from the attention calculation stage to the computing device through a cache consistency protocol; determining the task result of the model processing task based on the second result; wherein, the host connects the computing device and the memory processing device; and the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device through a cache consistency protocol.

[0007] In a third aspect, the present invention provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the aforementioned disclosed data processing method.

[0008] In a fourth aspect, the present invention provides a non-volatile storage medium for storing a computer program, wherein the computer program implements the aforementioned disclosed data processing method when executed by a processor.

[0009] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the aforementioned disclosed data processing method when executed by a processor.

[0010] It can be seen from the above scheme that the present invention provides a data processing system, including: a host, a computing device and a memory processing device; the host is connected to the computing device and the memory processing device; the cache consistency root port of the computing device is connected to the cache consistency subport of the memory processing device through a cache consistency protocol; the host is used to: send a model processing task to the computing device; the computing device is used to: execute the pre-filling stage and the query matrix and key value matrix calculation stage of the model processing task, and synchronize the first result output by the query matrix and key value matrix calculation stage to the memory processing device through the cache consistency protocol; the memory processing device is used to: execute the attention calculation stage of the model processing task based on the first result, and synchronize the second result output by the attention calculation stage to the computing device through the cache consistency protocol; the computing device is used to: determine the task result of the model processing task based on the second result.

[0011] It can be seen that the beneficial effects of the present invention are: the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device through the cache consistency protocol, so that the computing device is used to: synchronize the first result output by the query matrix and key-value matrix calculation phase of the model processing task to the memory processing device through the cache consistency protocol; at the same time, the memory processing device can synchronize the second result to the computing device through the cache consistency protocol, thereby realizing fast data transmission between the computing device and the memory processing device, which not only saves the memory space of the computing device, but also reduces the host processor overhead during the execution of the model processing task, and can improve the task processing efficiency of the hardware computing device to a certain extent.

[0012] Correspondingly, a data processing method, device, medium and program product provided by the present invention also have the above technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0014] Figure 1 A schematic diagram of a data processing system disclosed in the present invention; Figure 2 This is a schematic diagram of a second data processing system disclosed in the present invention; Figure 3 A schematic diagram of the GPU structure of a computing device disclosed in the present invention; Figure 4 This is a structural diagram of a memory processing device disclosed in the present invention; Figure 5 This is a schematic diagram of a Transformer-based large-model reasoning calculation disclosed in the present invention; Figure 6 for Figure 5 The calculation process shown is applied to the schematic diagram of the present invention; Figure 7 A server structure diagram provided by the present invention; Figure 8 This is a terminal structure diagram provided by the present invention. DETAILED DESCRIPTION

[0015] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making any creative efforts shall fall within the scope of protection of the present invention.

[0016] It should be noted that, in the description of the present invention, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. The terms "first," "second," etc., in the present invention are used to distinguish similar objects, and are not used to describe a particular order or precedence.

[0017] In order to enable those skilled in the art to better understand the solutions of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0018] Currently, when using hardware devices to perform tasks such as model reasoning, due to the limited memory space in the hardware devices, task-related data can be stored in the host. However, this requires the hardware device to frequently interact with the host during the execution of the model reasoning task, increasing the host processor overhead and reducing the task processing efficiency of the hardware device. To this end, the present invention provides a data processing solution that can achieve fast data transmission between a computing device and a memory processing device, which not only saves the memory space of the computing device, but also reduces the host processor overhead during the execution of the model processing task, and can improve the task processing efficiency of the hardware computing device to a certain extent.

[0019] See also Figure 1 As shown, an embodiment of the present invention discloses a data processing system comprising: a host, a computing device, and a memory processing device; the host is connected to the computing device and the memory processing device; a cache coherence root port of the computing device is connected to a cache coherence subport of the memory processing device via a cache coherence protocol. The computing device may be a GPU, FPGA, or the like. The host may be connected to the computing device and the memory processing device via a high-speed serial computer expansion bus (Peripheral Component Interconnect Express (PCIe)) and / or a cache coherence protocol. Furthermore, the host communicates with the computing device and the memory processing device using direct memory access (DMA or RDMA).

[0020] In this embodiment, the cache coherence protocol can use Compute Express Link (CXL). CXL is a high-speed interface protocol that optimizes the interaction between compute, storage, and communication resources within data centers. CXL consists of three sub-protocols: CXL.io, CXL.cache, and CXL.mem. CXL.io is used for initialization, linking, device identification and enumeration, and register access, providing a non-coherent load / store interface for devices. CXL.cache is used for cache access and defines the interaction between processors and devices, allowing connected CXL devices to cache processor memory efficiently with extremely low latency using a request and response method. CXL.mem is used for memory access, using load and store commands to provide processor access to device-attached memory. The processor acts as the master and the CXL device as the slave, supporting both volatile and persistent memory architectures. These protocols are dynamically multiplexed together to enable data transmission at 32GT / s over the standard PCIe 5.0 physical layer.

[0021] The host is used to send model processing tasks to the computing device. The model can be based on the Transformer architecture. Accordingly, the model processing tasks can include Transformer-based model training tasks, Transformer-based model inference tasks, etc. More specifically, the model processing tasks can include Transformer-based translation tasks, Transformer-based question-answering tasks, etc.

[0022] The computing device is used to perform the pre-population phase of the model processing task and the query matrix and key-value matrix calculation phase, synchronizing the first result output from the query matrix and key-value matrix calculation phase to the memory processing device via a cache coherence protocol. The pre-population phase provides preliminary foundational operations for the subsequent attention calculation phase to generate the Q, K, and V matrices, where Q is the query matrix and K and V constitute the KV key-value matrix.

[0023] The memory processing device is used to: execute the attention calculation phase of the model processing task based on the first result, and synchronize the second result (i.e., the O vector) output by the attention calculation phase to the computing device through a cache consistency protocol.

[0024] It should be noted that the memory processing device supports both memory and computing functions. The memory processing device stores the first result and performs the attention calculation phase. This can shorten the physical distance of data transmission and reduce data migration between devices compared to the traditional processor-memory device architecture. Therefore, the memory processing device can also be called a near-memory processing device, which offloads part of the computing task of the computing device to the memory device. Data can be directly retrieved and processed within the memory processing device. On the one hand, the path from data retrieval to data processing is greatly shortened, and on the other hand, cross-device data migration of the memory bus is reduced, alleviating the memory wall problem of the traditional architecture. If the memory device stores data and the GPU undertakes all calculations for the model processing task, the calculation process needs to frequently retrieve data from the memory device through the memory bus, resulting in high pressure on the memory bus bandwidth and the easy formation of a memory wall problem.

[0025] In one embodiment, the computing device is configured to use the three 1×d-dimensional matrices (i.e., the Q, K, and V matrices are 1×d-dimensional matrices) output from the query matrix and key-value matrix calculation phases as the first result. The memory processing device is configured to use the 1×d-dimensional fusion vector (i.e., the O matrix is ​​a 1×d-dimensional fusion vector) output from the attention calculation phase as the second result. This reduces the total amount of data required to be transferred between the computing device and the memory processing device, reducing both the total amount of data transferred during task execution and the size of each data transfer.

[0026] The computing device is used to determine a task result of the model processing task according to the second result.

[0027] It should be noted that the model processing task includes: pre-filling stage, query matrix and key value matrix calculation stage and attention calculation stage. Figure 2 The decoding phase PartA) and the attention calculation phase (i.e. Figure 2 The decoding stage Part B) in the decoding stage belongs to the decoding stage.

[0028] In one embodiment, the computing device includes: a first computing core and a first memory; the first computing core is connected to the first memory and a cache coherence root port (ie Figure 2 The first computing core is used to execute the pre-filling phase and the query matrix and key-value matrix calculation phase; and the first memory is used to store the input data of the model processing task, the results output by the pre-filling phase, and the first result.

[0029] Accordingly, the memory processing device includes: a second computing core and a second memory; the second computing core is connected to the second memory and the cache consistency subport (ie Figure 2The second CXL port in the cache coherence subport is configured to: (a) perform the attention calculation phase; (b) store the first result and the second result in the second memory; and (c) respond to requests from the memory controller using a double data rate (DDR) synchronous dynamic random access memory (DRAM).

[0030] In this embodiment, a computing device includes a memory request processing unit; the memory request processing unit includes a prefetch queue and a load queue; the prefetch queue is configured to store the addresses of requests in the load queue; and the load queue is configured to store requests to be loaded. Accordingly, the memory request processing unit includes a store queue; the store queue is configured to store the addresses of requests in the load queue and / or the addresses of requests in the prefetch queue. Specifically, the computing device is configured to read the second result from the memory processing device using a prefetch strategy.

[0031] In one embodiment, the second computing core includes: a matrix calculation unit for implementing the matrix calculation included in the attention calculation stage; a vector calculation unit for implementing the vector calculation included in the attention calculation stage; a buffer controller for managing the buffer; a direct memory access controller for implementing data interaction between the buffer and the second memory; and a buffer for storing the calculation commands included in the attention calculation stage.

[0032] In one embodiment, the host is used to: scan the computing device and the memory processing device after startup, initialize the computing device and the memory processing device, and allocate address space to the computing device and the memory processing device.

[0033] In one embodiment, both the cache coherence root port and the cache coherence subport have built-in cache coherence controllers. The cache coherence controllers are configured to perform data processing at the physical layer, data link layer, and transaction layer. Specifically, the cache coherence controllers are configured to implement physical information exchange, interface initialization, and maintenance at the physical layer, flow control and link state management at the data link layer, and packet encapsulation and parsing at the transaction layer.

[0034] In this embodiment, the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device through the cache consistency protocol, so that the computing device is used to: synchronize the first result output by the query matrix and key-value matrix calculation phase of the model processing task to the memory processing device through the cache consistency protocol; at the same time, the memory processing device can synchronize the second result to the computing device through the cache consistency protocol, thereby realizing fast data transmission between the computing device and the memory processing device, which not only saves the memory space of the computing device, but also reduces the host processor overhead during the execution of the model processing task, and can improve the task processing efficiency of the hardware computing device to a certain extent.

[0035] See Figure 2 Another data processing system includes a host CPU, a computing device GPU, and a CXL-PNM (CXL-Processing Near Memory). The host CPU, the computing device GPU, and the CXL-PNM are connected via a PCIe or CXL bus. The GPU and PNM communicate using the CXL protocol.

[0036] (1) On the CPU side, the firmware is responsible for scanning the GPU and PNM devices after power-on, unifying address space allocation, and initializing registers such as the HDM decoder in the GPU and PNM devices. The data path between the CPU and the two devices can be used for device configuration and task initialization, as well as for large-scale data transfer using PCIe DMA.

[0037] (2) The GPU can be implemented using the open-source Vortex architecture and is responsible for the main computational tasks of the pre-filling and decoding phases of LLM inference (Part A). Data outside the KV Cache, including weights and activation functions, is stored in on-chip video memory such as VRAM. Furthermore, to support CXL, the first CXL port (RP) must be integrated on the GPU.

[0038] (3) The CXL-PNM device is primarily responsible for implementing KV Cache storage, updates, and access retrieval, as well as the attention mechanism calculation during the decoding phase (Part B) that frequently accesses the KV Cache. At the same time, to support CXL, a second CXL port, CXL EP (Endpoint Port), needs to be integrated on the PNM device.

[0039] See Figure 3 , the main implementations in GPU are: (1) The CXL controller, which implements the CXL protocol, has a three-layer architecture. The Flex bus physical layer is responsible for physical information exchange, interface initialization, and maintenance. The data link layer provides reliable data transmission services, including link status and management transactions such as flow control and credit. The transaction layer handles transactions associated with each protocol and performs packet encapsulation and parsing. In the GPU, the CXL controller is instantiated in CXL RP mode.

[0040] (2) The communication computing unit of the Vortex architecture GPU is a streaming multiprocessor (multiple SMs), which is connected to the AXI or ACE system bus through the last level cache (LLC, third level cache). In addition, the computing unit is connected to the local memory controller and PCIe port (PCIe EP) through the AXI / ACE system bus to meet memory access requirements and communication requirements with the host. After the host completes the system scan and enumeration, it configures the address decoder and related physical address range registers through PCIe bus transactions. The address decoder receives the address code sent by the computing unit and decodes it, outputting the enable signal for the on-chip memory device or the extended video memory hardware device.

[0041] (3) Since the latency of CXL extended memory is larger than that of GPU local memory, and considering that the CXL protocol transaction layer supports speculative read transactions, a data prefetch strategy can be designed in the memory request hardware unit, and a prefetch queue and a load queue buffer can be designed. If the load request address is included in the prefetch queue, the two requests are merged into one load request, and the request in the load queue has a higher priority than the prefetch request. The MUX arbitrator performs reasonable arbitration based on the traffic of the two queues and sends a request to the CXL controller transaction layer. In addition, a store-to-load forwarding strategy is designed, that is, if the address of the load or prefetch request is in the current store queue, the data can be directly retrieved from the store queue and the load data result can be returned. Through the prefetch strategy and store-to-load forwarding strategy, the access performance of the extended memory can be effectively improved.

[0042] See Figure 4 , CXL-PNM can be implemented based on FPGA, mainly including: (1) The CXL controller is instantiated in CXL EP mode to implement the CXL protocol parsing and response functions on the device side. In addition, the PNM device also needs to instantiate an upstream port PCIe / CXL EP for connecting to the host ( Figure 4 The host can initialize and control the device and scan and configure the Configuration Space Registers (CSRs) through PCIe transactions.

[0043] (2) The Key and Value caches during the LLM inference process are stored in the DDR of the NPM device. The PNM computing unit implements a small number of computational steps related to KV value access (QKV calculation) in the inference calculation. This is implemented by the PNM acceleration computing unit on the device, which mainly includes: matrix computing unit, vector computing unit, buffer controller, buffer, DMA controller and other modules. The buffer stores load / store instructions and the KV data required to execute the computation instructions. The DMA controller controls the batch data transmission between the buffer and the device DDR.

[0044] (3) The CXL.mem request processing unit receives the parsed CXL.mem request (from the GPU) from the CXL controller, accesses the HDM decoder register in the configuration space, translates it into a local address, and then issues a DDR memory access request to the memory controller (MC). The memory request arbiter arbitrates the requests from the PNM compute unit and CXL.mem and forwards them to the MC memory controller for read and write data operations. The CXL.mem request is passed to the CXL controller via the Flex bus physical layer, arbitration, and multiplexer CXLARB / MUX.

[0045] If the model processing task is Transformer-based model inference, the following example uses Transformer-based model inference as an example. Given an input sequence, it first goes through a pre-filling phase and then through multiple decoding phases, generating output tokens one by one. The output token of each decoding phase serves as the input for the next decoding phase, and this process repeats until the end token is generated.

[0046] See Figure 5 Each pre-filling or decoding phase follows a similar process: the input sequence is converted into an embedding vector through the embedding layer, then computed through N cascaded decoding layers to generate an output embedding vector, which is then converted into an output token. The decoding layer has two key sublayers: the self-attention layer and the feed-forward network (FFN) layer. The self-attention layer consists of a QKV generation layer, an attention calculation layer, and a linear mapping layer (fully connected layer) of the attention layer output. The attention calculation mainly involves: d and k represent the embedding dimensions of the matrix K.

[0047] It should be noted that each pre-filling or decoding phase follows the following calculation process: the input sequence is converted into an embedding vector through the embedding layer, then passed through N cascaded decoding layers to generate an output embedding vector, which is then converted into an output token. The decoding layer has two key sublayers: the self-attention layer and the feed-forward network (FFN) layer. The self-attention layer consists of a QKV generation layer, an attention calculation layer, and a linear mapping layer (fully connected layer) of the attention layer output. The decoding phase, Part A, consists of an embedding layer, a decoding layer (FFN, a QKV generation layer for self-attention calculation, and a linear mapping layer of the attention layer output). The embedding layer maps discrete word IDs into a continuous vector space; the QKV generation layer uses linear mapping to generate query, key, and value vectors; the attention layer performs a weighted combination of these vectors to achieve information fusion; and the FFN stage uses nonlinear mapping to further abstract feature representation.

[0048] Accordingly, Figure 5 The hardware and data interaction processes required for calculation in each stage are shown as follows Figure 6 Shown, including: (1) The calculation of the pre-filling stage is performed on the GPU. The attention layer calculation generates a key-value (KV) matrix for the subsequent decoding stage, which is the KV Cache. It is stored in the DDR memory of the PNM via DMA batch transfer. The size of the K matrix and the V matrix are both s0×d (s0 is the input sequence length and d is the embedding dimension).

[0049] (2) The input of the decoding stage is a single token generated in the previous stage, and the input of each decoder is also a vector. The attention calculation mainly consists of GEMV operations, which use the q vector of the current input token and the aggregated KV matrix to perform calculations on the PNM computing device shown above. Regarding the aggregated KV matrix, specifically, each time a new token is decoded, the corresponding q, k, and v vectors are generated, where the k and v vectors are attached to the KV matrix passed down from the previous stage in real time, forming a growing aggregated KV matrix in the PNM memory, as shown in the following example: Figure 5 This is shown in the Attention mechanism box on the far right of the figure. During this process, the q vector has a dimension of 1×d, and the K and V matrices have dimensions of s×d, where s represents the sum of s0 and the number of tokens generated in the current stage.

[0050] Accordingly, this embodiment has the following advantages: (1) The attention calculation is offloaded to PNM. Unlike the original reading of the entire s×d dimension KV matrix, only the current 1×d dimension q, k, v matrix and output o matrix are transmitted on the inter-card interconnection bus, which greatly reduces the dimension of the transmitted data (originally transmitting the complete KV, now transmitting one-dimensional). This advantage is more obvious under the trend of long sequence reasoning. (2) Regarding the data transmission granularity, taking GPT-3 as an example, a 96-head multi-head attention mechanism is adopted. If the single-head embedding dimension is 128 and FP16 quantization is used, the size of the q / k / v / o matrix of each head is 256B; if the single-head embedding dimension is 64 and INT8 quantization is used, the size of the q / k / v / o matrix of each head is 64B. In the decoding stage, each decoder decoder will frequently write the q, k, v vectors and read the o vector, and data transmission is relatively frequent. The CXL-based extended video memory solution, with the help of the CXL bus hardware to maintain data consistency, implicitly and automatically synchronizes the updated QKV vectors to the extended video memory, achieving efficient synchronization of fine-grained data (synchronization between the GPU cache and the DDR in the PNM). This can significantly reduce transmission latency and software-level synchronization overhead, allowing data transmission to be efficiently overlapped with the calculation stage, thereby shortening the overall inference time.

[0051] As can be seen, this embodiment implements KV Cache offloading for GPU graphics cards, significantly alleviating KV Cache storage pressure. Specifically, the KV Cache, which requires frequent access during the decoding phase, is offloaded to the GPU graphics card's extended video memory (PNM), reducing the frequent transmission of KV matrices on the bus, improving communication efficiency, and effectively enhancing the overall performance of large-model inference. CXL interconnect technology supports automatic implicit hardware updates of computational data. The KV tensors generated token by token in the GPU during the decoding phase are automatically updated to the extended video memory via the CXL bus with hardware-maintained consistency. This significantly reduces software overhead, allows for efficient overlap with the computation phase, and shortens overall inference time.

[0052] A data processing method provided by an embodiment of the present invention is introduced below. The data processing method described below can be referenced with other embodiments described herein.

[0053] An embodiment of the present invention discloses a data processing method, which is applied to a computing device in a data processing system, including: executing a pre-filling phase and a query matrix and key-value matrix calculation phase of a model processing task sent by a host in the data processing system; synchronizing a first result output from the query matrix and key-value matrix calculation phase to a memory processing device in the data processing system through a cache consistency protocol, so that the memory processing device executes the attention calculation phase of the model processing task based on the first result, and synchronizing a second result output from the attention calculation phase to the computing device through a cache consistency protocol; and determining a task result of the model processing task based on the second result.

[0054] The host is connected to the computing device and the memory processing device; the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device through a cache consistency protocol.

[0055] In one embodiment, the computing device includes: a first computing core and a first memory; the first computing core is connected to the first memory and a cache consistency root port; the first computing core is used to: execute a pre-filling phase and a query matrix and key-value matrix calculation phase; the first memory is used to: store input data of a model processing task, a result outputted from the pre-filling phase, and a first result.

[0056] In one embodiment, the memory processing device includes: a second computing core and a second memory; the second computing core is connected to the second memory and the cache consistency subport; the second computing core is used to: execute the attention calculation stage; the second memory is used to: store the first result and the second result.

[0057] In one embodiment, the second memory includes: a memory controller and a storage medium; the memory controller is used to: parse the synchronization request from the cache consistency subport and access the storage medium according to the parsing result; the storage medium is used to: respond to the access of the memory controller.

[0058] In one embodiment, the storage medium is a double data rate synchronous dynamic random access memory.

[0059] In one embodiment, a computing device includes a memory request processing unit, which includes a prefetch queue and a load queue. The prefetch queue is used to store addresses of requests in the load queue. The load queue is used to store requests to be loaded.

[0060] In one embodiment, the memory request processing unit includes a storage queue; the storage queue is used to store the address of each request in the load queue and / or store the address of each request in the prefetch queue.

[0061] In one embodiment, the second computing core includes: a matrix calculation unit for implementing the matrix calculation included in the attention calculation stage; a vector calculation unit for implementing the vector calculation included in the attention calculation stage; a buffer controller for managing the buffer; a direct memory access controller for implementing data interaction between the buffer and the second memory; and a buffer for storing the calculation commands included in the attention calculation stage.

[0062] In one embodiment, the host connects the computing device and the memory processing device via a high-speed serial computer expansion bus and / or a cache coherence protocol.

[0063] In one embodiment, the host communicates with the computing device and the memory processing device respectively using direct memory access technology.

[0064] In one embodiment, the host is used to: scan the computing device and the memory processing device after startup, initialize the computing device and the memory processing device, and allocate address space to the computing device and the memory processing device.

[0065] In one embodiment, both the cache coherence root port and the cache coherence sub-port have a built-in cache coherence controller; the cache coherence controller is used to perform corresponding data processing at the physical layer, the data link layer, and the transaction layer respectively.

[0066] In one embodiment, the cache coherence controller is used to implement physical information exchange, interface initialization and maintenance at the physical layer, flow control and link state management at the data link layer, and data packet encapsulation and parsing at the transaction layer.

[0067] In one embodiment, the computing device is configured to: use the three 1×d-dimensional matrices outputted from the query matrix and key-value matrix computing phase as the first result.

[0068] In one embodiment, the memory processing device is used to: use the 1×d-dimensional fusion vector output by the attention calculation stage as the second result.

[0069] In one embodiment, the computing device is configured to read the second result from the memory processing device using a prefetch strategy.

[0070] Among them, for more specific working processes of each module and unit in this embodiment, reference can be made to the corresponding contents disclosed in the aforementioned embodiments, which will not be repeated here.

[0071] It can be seen that this embodiment provides a data processing device that can realize fast data transmission between a computing device and a memory processing device, which not only saves the memory space of the computing device, but also reduces the host processor overhead during the execution of the model processing task, and can improve the task processing efficiency of the hardware computing device to a certain extent.

[0072] The following describes an electronic device provided by an embodiment of the present invention. The electronic device described below can be referenced with other embodiments described herein. The electronic device in this embodiment can be a host, a computing device, or a memory processing device, or any functional module in these devices.

[0073] An embodiment of the present invention discloses an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method disclosed in any of the above embodiments.

[0074] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: execute the pre-filling stage and the query matrix and key value matrix calculation stage of the model processing task sent by the host in the data processing system; synchronize the first result output by the query matrix and key value matrix calculation stage to the memory processing device in the data processing system through the cache consistency protocol.

[0075] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: execute the attention calculation phase of the model processing task based on the first result, and synchronize the second result output by the attention calculation phase to the computing device through the cache consistency protocol.

[0076] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: determining the task result of the model processing task according to the second result.

[0077] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: executing a pre-filling phase and a query matrix and key-value matrix calculation phase.

[0078] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the input data of the model processing task, the results output in the pre-filling stage, and the first result.

[0079] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: executing the attention calculation stage.

[0080] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: storing the first result and the second result.

[0081] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: parsing the synchronization request from the cache coherence subport, and accessing the storage medium according to the parsing result.

[0082] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: responding to access from the memory controller.

[0083] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: storing the address of each request in the load queue.

[0084] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: storing the request to be loaded.

[0085] In this embodiment, when the processor executes the computer program stored in the memory, it may specifically implement the following steps: storing the address of each request in the load queue, and / or storing the address of each request in the prefetch queue.

[0086] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: implement the matrix calculation included in the attention calculation stage.

[0087] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: implement the vector calculation included in the attention calculation stage.

[0088] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: managing a buffer zone.

[0089] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: data interaction between the buffer and the second memory is implemented.

[0090] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the calculation commands included in the attention calculation stage.

[0091] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: communicating with the computing device and the memory processing device respectively using direct memory access technology.

[0092] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: after startup, scan the computing device and the memory processing device, initialize the computing device and the memory processing device, and allocate address space to them.

[0093] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: performing corresponding data processing at the physical layer, the data link layer, and the transaction layer respectively.

[0094] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: implement physical information exchange, interface initialization and maintenance at the physical layer, implement flow control and link status management at the data link layer, and implement data packet encapsulation and parsing at the transaction layer.

[0095] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: taking the three 1×d-dimensional matrices outputted from the query matrix and key value matrix calculation phase as the first result.

[0096] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: using the 1×d-dimensional fusion vector outputted in the attention calculation stage as the second result.

[0097] In this embodiment, when the processor executes the computer program stored in the memory, the following steps may be specifically implemented: reading the second result from the memory processing device using a prefetch strategy.

[0098] Furthermore, an embodiment of the present invention also provides an electronic device. Figure 7 The server shown can also be Figure 8 Terminal shown. Figure 7 and Figure 8 Each of the diagrams is a structural diagram of an electronic device according to an exemplary embodiment, and the contents in the diagrams should not be considered as limiting the scope of application of the present invention.

[0099] Figure 7 This is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server may include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory is used to store a computer program, which is loaded and executed by the processor to implement the relevant steps of the data processing disclosed in any of the aforementioned embodiments.

[0100] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices. The communication protocol it follows is any communication protocol that can be applied to the technical solution of the present invention and is not specifically limited here; the input and output interface is used to obtain external input data or output data to the outside world. The specific interface type can be selected according to specific application needs and is not specifically limited here.

[0101] In addition, the memory as a carrier for resource storage can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon include operating system, computer programs and data, etc. The storage method can be temporary storage or permanent storage.

[0102] The operating system is used to manage and control the hardware devices and computer programs on the server, enabling the processor to operate and process data in the memory. It can be Windows Server, NetWare, Unix, Linux, etc. In addition to computer programs capable of performing the data processing methods disclosed in any of the aforementioned embodiments, computer programs can also include computer programs capable of performing other specific tasks. Data can include data such as application update information and other data such as application developer information.

[0103] Figure 8 This is a schematic diagram of the structure of a terminal provided by an embodiment of the present invention. The terminal may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0104] Generally, the terminal in this embodiment includes: a processor and a memory.

[0105] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor may be implemented in at least one of the following hardware forms: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). The processor may also include a main processor and a coprocessor. The main processor is used to process data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing content required to be displayed on the display. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0106] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory, and non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory is used to store at least the following computer program, wherein, after the computer program is loaded and executed by the processor, it can implement the relevant steps in the data processing method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory may also include an operating system and data, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system may include Windows, Unix, Linux, etc. The data may include but is not limited to update information of the application.

[0107] In some embodiments, the terminal may further include a display screen, an input and output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0108] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation to the terminal, and may include more or fewer components than shown in the figure.

[0109] A non-volatile storage medium provided in an embodiment of the present invention is introduced below. The non-volatile storage medium described below can be referenced with other embodiments described herein.

[0110] A non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method disclosed in the aforementioned embodiment. The non-volatile storage medium is a computer-readable non-volatile storage medium that, as a carrier for resource storage, may be a read-only memory, random access memory, a magnetic disk, or an optical disk. The resources stored thereon include an operating system, a computer program, and data, and the storage method may be either temporary or permanent.

[0111] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: execute the pre-filling stage and the query matrix and key-value matrix calculation stage of the model processing task sent by the host in the data processing system; synchronize the first result output from the query matrix and key-value matrix calculation stage to the memory processing device in the data processing system through the cache consistency protocol.

[0112] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: execute the attention calculation phase of the model processing task based on the first result, and synchronize the second result output by the attention calculation phase to the computing device through the cache consistency protocol.

[0113] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: determining the task result of the model processing task according to the second result.

[0114] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: executing a pre-filling phase and a query matrix and key-value matrix calculation phase.

[0115] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the input data of the model processing task, the result outputted in the pre-filling phase, and the first result.

[0116] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: executing the attention calculation phase.

[0117] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: storing the first result and the second result.

[0118] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: parsing the synchronization request from the cache coherence subport, and accessing the storage medium according to the parsing result.

[0119] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: responding to access from the memory controller.

[0120] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: storing the address of each request in the load queue.

[0121] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: storing the request to be loaded.

[0122] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it may specifically implement the following steps: storing the address of each request in the load queue, and / or storing the address of each request in the prefetch queue.

[0123] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: implement the matrix calculation included in the attention calculation stage.

[0124] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: implement the vector calculation included in the attention calculation stage.

[0125] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: managing a buffer zone.

[0126] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: realizing data interaction between the buffer and the second memory.

[0127] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the calculation commands included in the attention calculation stage.

[0128] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: communicating with the computing device and the memory processing device respectively using direct memory access technology.

[0129] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: after startup, scan the computing device and memory processing device, initialize the computing device and memory processing device, and allocate address space to them.

[0130] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: performing corresponding data processing at the physical layer, the data link layer, and the transaction layer, respectively.

[0131] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: implement physical information exchange, interface initialization and maintenance at the physical layer, implement flow control and link status management at the data link layer, and implement data packet encapsulation and parsing at the transaction layer.

[0132] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: three 1×d-dimensional matrices outputted from the query matrix and key value matrix calculation phase are used as the first result.

[0133] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: using the 1×d-dimensional fusion vector output by the attention calculation stage as the second result.

[0134] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, the following steps may be specifically implemented: reading the second result from the memory processing device using a prefetch strategy.

[0135] A computer program product provided by an embodiment of the present invention is introduced below. The computer program product described below can be referenced with other embodiments described herein.

[0136] A computer program product comprises a computer program / instruction, which implements the steps of the aforementioned data processing method when executed by a processor.

[0137] An embodiment of the present invention further provides another computer program product, including a non-volatile computer-readable storage medium, where the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the steps in any of the above embodiments are implemented.

[0138] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0139] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-volatile storage medium known in the art.

[0140] Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only applicable to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A data processing system, characterized in that: include: Hosts, computing devices, and memory processing devices; The host is connected to the computing device and the memory processing device; The cache coherence root port of the computing device is connected to the cache coherence subport of the memory processing device via a cache coherence protocol; The host is used to: send the model processing task to the computing device; The computing device is used to: execute the pre-filling phase and the query matrix and key value matrix calculation phase of the model processing task, and synchronize the first result output by the query matrix and key value matrix calculation phase to the memory processing device through a cache consistency protocol; The memory processing device is used to: perform the attention calculation phase of the model processing task based on the first result, and synchronize the second result output by the attention calculation phase to the computing device through a cache consistency protocol; The computing device is used to determine a task result of the model processing task according to the second result.

2. The system according to claim 1, wherein: The computing device comprises: a first computing core and a first memory; the first computing core is connected to the first memory and the cache coherence root port; The first computing core is used to: execute the pre-filling phase and the query matrix and key value matrix calculation phase; The first memory is used to store input data of the model processing task, the result output by the pre-filling stage and the first result.

3. The system according to claim 1, wherein: The memory processing device includes: a second computing core and a second memory; the second computing core is connected to the second memory and the cache consistency subport; The second computing core is used to: execute the attention calculation stage; The second memory is used to store the first result and the second result.

4. The system according to claim 3, characterized in that The second memory includes: a memory controller and a storage medium; The memory controller is used to: parse the synchronization request from the cache coherence subport and access the storage medium according to the parsing result; The storage medium is used to respond to the access of the memory controller.

5. The system according to claim 4, characterized in that The storage medium is a double rate synchronous dynamic random access memory.

6. The system according to claim 2, wherein: The computing device includes: a memory request processing unit; the memory request processing unit includes: a prefetch queue and a load queue; The prefetch queue is used to: store the addresses of each request in the load queue; The loading queue is used to store requests to be loaded.

7. The system according to claim 6, characterized in that The memory request processing unit includes: a storage queue; the storage queue is used to: store the address of each request in the load queue and / or store the address of each request in the prefetch queue.

8. The system according to claim 3, wherein: The second computing core includes: The matrix calculation unit is used to: implement the matrix calculation included in the attention calculation stage; The vector calculation unit is used to: implement the vector calculation included in the attention calculation stage; The buffer controller is used to: manage the buffer; The direct memory access controller is used to: implement data interaction between the buffer and the second memory; The buffer is used to store the calculation commands included in the attention calculation stage.

9. The system according to claim 1, wherein: The host is connected to the computing device and the memory processing device via a high-speed serial computer expansion bus and / or a cache coherence protocol.

10. The system according to claim 9, characterized in that The host communicates with the computing device and the memory processing device respectively using direct memory access technology.

11. The system according to claim 1, wherein: The host is used to scan the computing device and the memory processing device after startup, and initialize and allocate address spaces to the computing device and the memory processing device.

12. The system according to claim 1, wherein: The cache coherence root port and the cache coherence sub-port both have a built-in cache coherence controller; The cache consistency controller is used to perform corresponding data processing at the physical layer, data link layer and transaction layer respectively.

13. The system according to claim 12, wherein: The cache consistency controller is used to: implement physical information exchange, interface initialization and maintenance at the physical layer, implement flow control and link state management at the data link layer, and implement data packet encapsulation and analysis at the transaction layer.

14. The system according to any one of claims 1 to 13, characterized in that The computing device is used to: use the three 1×d-dimensional matrices outputted in the query matrix and key-value matrix computing phase as the first result.

15. The system according to any one of claims 1 to 13, characterized in that The memory processing device is used to: use the 1×d dimensional fusion vector output by the attention calculation stage as the second result.

16. The system according to any one of claims 1 to 13, characterized in that The computing device is configured to read the second result from the memory processing device using a prefetch strategy.

17. A data processing method, characterized in that: Computing devices used in data processing systems, including: Executing the pre-filling phase and the query matrix and key-value matrix calculation phase of the model processing task sent by the host in the data processing system; Synchronizing a first result output from the query matrix and key-value matrix calculation phase to a memory processing device in the data processing system through a cache consistency protocol, so that the memory processing device performs the attention calculation phase of the model processing task based on the first result, and synchronizing a second result output from the attention calculation phase to the computing device through a cache consistency protocol; Determining a task result of the model processing task according to the second result; The host is connected to the computing device and the memory processing device; the cache consistency root port of the computing device is connected to the cache consistency subport of the memory processing device through a cache consistency protocol.

18. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to execute the computer program to implement the method according to claim 17.

19. A non-volatile storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the method according to claim 17 is implemented.

20. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method of claim 17 is implemented.

Images