A data processing system, method, device, medium, and program product

Abstract

This invention discloses a data processing system, method, device, medium, and program product in the field of computer technology. In this invention, the cache coherence root port of the computing device is connected to the cache coherence sub-port of the memory processing device via a cache coherence protocol. This enables the computing device to: synchronize the first result output from the query matrix and key-value matrix calculation stages of the model processing task to the memory processing device via the cache coherence protocol; simultaneously, the memory processing device can synchronize the second result to the computing device via the cache coherence protocol. This achieves fast data transfer between the computing device and the memory processing device, saving memory space on the computing device and reducing the host processor overhead during the execution of the model processing task, thereby improving the task processing efficiency of the hardware computing device to a certain extent.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a data processing system, method, device, medium, and program product. Background Technology

[0002] Currently, when using hardware devices to perform tasks such as model inference, due to the limited memory space in the hardware devices, task-related data can be stored in the host. However, this requires the hardware devices to frequently interact with the host during the model inference process, increasing the overhead of the host processor and reducing the task processing efficiency of the hardware devices.

[0003] Therefore, how to improve the efficiency of hardware devices in performing tasks is a problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of the present invention is to provide a data processing system, method, device, medium and program product to improve the efficiency of hardware devices in performing tasks.

[0005] In a first aspect, the present invention provides a data processing system, comprising: a host, a computing device, and a memory processing device; the host is connected to the computing device and the memory processing device; the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device via a cache consistency protocol; the host is used to: send a model processing task to the computing device; the computing device is used to: execute a pre-filling stage and a query matrix and key-value matrix calculation stage of the model processing task, and synchronize the first result output by the query matrix and key-value matrix calculation stage to the memory processing device via a cache consistency protocol; the memory processing device is used to: execute an attention calculation stage of the model processing task based on the first result, and synchronize the second result output by the attention calculation stage to the computing device via a cache consistency protocol; the computing device is used to: determine the task result of the model processing task based on the second result.

[0006] Secondly, the present invention provides a data processing method applied to a computing device in a data processing system, comprising: executing a pre-filling stage and a query matrix and key-value matrix calculation stage of a model processing task sent by a host in the data processing system; synchronizing a first result output by the query matrix and key-value matrix calculation stage to a memory processing device in the data processing system via a cache consistency protocol, so that the memory processing device executes an attention calculation stage of the model processing task based on the first result; synchronizing a second result output by the attention calculation stage to the computing device via the cache consistency protocol; and determining the task result of the model processing task based on the second result; wherein, the host connects the computing device and the memory processing device; and the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device via a cache consistency protocol.

[0007] Thirdly, the present invention provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the aforementioned disclosed data processing method.

[0008] Fourthly, the present invention provides a non-volatile storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned disclosed data processing method.

[0009] Fifthly, the present invention provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed data processing method.

[0010] As can be seen from the above scheme, the present invention provides a data processing system, including: a host, a computing device, and a memory processing device; the host is connected to the computing device and the memory processing device; the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device through a cache consistency protocol; the host is used to: send model processing tasks to the computing device; the computing device is used to: execute the pre-filling stage and the query matrix and key-value matrix calculation stage of the model processing task, and synchronize the first result output by the query matrix and key-value matrix calculation stage to the memory processing device through the cache consistency protocol; the memory processing device is used to: execute the attention calculation stage of the model processing task based on the first result, and synchronize the second result output by the attention calculation stage to the computing device through the cache consistency protocol; the computing device is used to: determine the task result of the model processing task based on the second result.

[0011] As can be seen, the beneficial effects of the present invention are as follows: the cache coherence root port of the computing device is connected to the cache coherence sub-port of the memory processing device through the cache coherence protocol, so that the computing device can synchronize the first result output by the query matrix and key-value matrix calculation stage of the model processing task to the memory processing device through the cache coherence protocol; at the same time, the memory processing device can synchronize the second result to the computing device through the cache coherence protocol, thereby realizing fast data transmission between the computing device and the memory processing device, which not only saves the memory space of the computing device, but also reduces the host processor overhead during the execution of the model processing task, and can improve the task processing efficiency of the hardware computing device to a certain extent.

[0012] Correspondingly, the data processing method, device, medium, and program product provided by the present invention also have the above-mentioned technical effects. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of a data processing system disclosed in this invention;

[0015] Figure 2 This is a schematic diagram of the second data processing system disclosed in this invention;

[0016] Figure 3 This is a schematic diagram of a GPU structure for a computing device disclosed in this invention;

[0017] Figure 4 This is a schematic diagram of a memory processing device structure disclosed in this invention;

[0018] Figure 5 This is a schematic diagram of a large model inference computation based on Transformer disclosed in this invention;

[0019] Figure 6 for Figure 5 The diagram illustrates the application of the calculation process in this invention.

[0020] Figure 7 A server structure diagram provided by the present invention;

[0021] Figure 8 This invention provides a terminal structure diagram. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0023] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0024] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0025] Currently, when performing tasks such as model inference using hardware devices, the limited memory space in the hardware device necessitates storing task-related data on the host machine. However, this requires frequent interaction between the hardware device and the host machine during model inference, increasing the overhead on the host processor and reducing the hardware device's task processing efficiency. To address this, this invention provides a data processing solution that enables rapid data transfer between the computing device and the memory processing device. This saves memory space on the computing device and reduces the host processor overhead during model processing, thereby improving the task processing efficiency of the hardware computing device to some extent.

[0026] See Figure 1 As shown in the figure, this invention discloses a data processing system, including: a host, a computing device, and a memory processing device; the host is connected to the computing device and the memory processing device; the cache coherence root port of the computing device is connected to the cache coherence sub-port of the memory processing device through a cache coherence protocol. The computing device may be a GPU, FPGA, etc. The host can connect to the computing device and the memory processing device through a high-speed serial computer expansion bus (PCIe, Peripheral Component Interconnect express) and / or a cache coherence protocol. Furthermore, the host communicates with the computing device and the memory processing device respectively using direct memory access technology (DMA or RDMA).

[0027] In this embodiment, the cache coherency protocol can adopt CXL (Compute Express Link), a high-speed interface protocol that optimizes the interaction between computing, storage, and communication resources within a data center. CXL actually consists of three sub-protocols: CXL.io, CXL.cache, and CXL.mem. CXL.io is used for initialization, linking, device identification and enumeration, and register access, providing a non-consistent load / store interface for devices. CXL.cache is used for cache access, defining the interaction between the processor and the device, allowing connected CXL devices to use request and response methods to efficiently cache processor memory with extremely low latency. CXL.mem is used for memory access, using load and store commands to provide the processor with access to the device's attached memory, where the processor acts as the master device and the CXL device as the slave device, supporting both volatile and persistent memory architectures. These protocols are dynamically multiplexed together, enabling data transfer at a speed of 32GT / s through the standard PCIe 5.0 physical layer.

[0028] The host computer is used to send model processing tasks to the computing device. The model can be a Transformer-based model. Accordingly, the model processing tasks can be Transformer-based model training tasks, Transformer-based model inference tasks, etc. More specifically, the model processing tasks can be Transformer-based translation tasks, Transformer-based question answering tasks, etc.

[0029] The computing device is used to: perform the pre-filling phase and the query matrix and key-value matrix calculation phase of the model processing task; and synchronize the first results output from the query matrix and key-value matrix calculation phase to the memory processing device through a cache consistency protocol. The pre-filling phase provides the preliminary basic calculations for the subsequent attention calculation phase to generate the Q, K, and V matrices, where Q is the query matrix, and K and V constitute the KV key-value matrix.

[0030] The memory processing device is used to: perform the attention calculation phase of the model processing task based on the first result, and synchronize the second result (i.e., the O vector) output by the attention calculation phase to the computing device through a cache consistency protocol.

[0031] It's important to note that the memory processing device supports both memory and computation functions. By storing the initial results and performing the attention calculation phase on the memory processing device, the physical distance for data transmission is shortened, and compared to the traditional processor-memory device architecture, data migration between devices is reduced. Therefore, this memory processing device can also be called a near-memory processing device. This offloads some computational tasks from the computing device to the memory device, allowing data to be retrieved and processed directly within the memory processing device. This significantly reduces the path from data retrieval to processing and minimizes cross-device data migration on the memory bus, alleviating the memory wall problem of traditional architectures. If data is stored in the memory device and the GPU handles all the computation for model processing, frequent data retrieval from the memory device during computation puts high pressure on the memory bus bandwidth, easily leading to the memory wall problem.

[0032] In one implementation, the computing device is used to: take the three 1×d-dimensional matrices (i.e., Q, K, and V matrices are 1×d-dimensional matrices) output from the query matrix and key-value matrix calculation phase as the first result. The memory processing device is used to: take the 1×d-dimensional fusion vector (i.e., the O matrix is ​​a 1×d-dimensional fusion vector) output from the attention calculation phase as the second result. Thus, only 1×d-dimensional data needs to be transferred between the computing device and the memory processing device, reducing both the total amount of data transferred during task execution and the size of each transfer.

[0033] The computing device is used to: determine the task result of the model processing task based on the second result.

[0034] It should be noted that the model processing task includes: a pre-filling stage, a query matrix and key-value matrix calculation stage, and an attention calculation stage. The query matrix and key-value matrix calculation stage (i.e....) Figure 2 The decoding stage (Part A) and the attention computation stage (i.e. Figure 2 The decoding stage (Part B) in the text all belong to the decoding stage.

[0035] In one embodiment, the computing device includes: a first computing core and a first memory; the first computing core is connected to the first memory and a cache coherence root port (i.e., Figure 2 The first CXL port in the first computing core is used to: perform the pre-filling stage and the query matrix and key-value matrix calculation stage; the first memory is used to: store the input data of the model processing task, the output results of the pre-filling stage, and the first result.

[0036] Accordingly, the memory processing device includes: a second computing core and a second memory; the second computing core is connected to the second memory and a cache coherence subport (i.e., Figure 2The second computational core is used to perform the attention computation phase; the second memory is used to store the first result and the second result. The second memory includes a memory controller and a storage medium; the memory controller is used to parse synchronization requests from the cache coherence subport and access the storage medium based on the parsing result; the storage medium is used to respond to the access from the memory controller. The storage medium is a double data rate synchronous dynamic random access memory (DDR).

[0037] In this embodiment, the computing device includes a memory request processing unit; the memory request processing unit includes a prefetch queue and a load queue; the prefetch queue is used to store the addresses of each request in the load queue; the load queue is used to store requests to be loaded. Correspondingly, the memory request processing unit includes a storage queue; the storage queue is used to store the addresses of each request in the load queue, and / or store the addresses of each request in the prefetch queue. Specifically, the computing device is used to read a second result from the memory processing device according to a prefetch strategy.

[0038] In one embodiment, the second computing core includes: a matrix computing unit for performing matrix calculations included in the attention computing stage; a vector computing unit for performing vector calculations included in the attention computing stage; a buffer controller for managing a buffer; a direct memory access controller for enabling data interaction between the buffer and a second memory; and a buffer for storing computing commands included in the attention computing stage.

[0039] In one implementation, the host is used to: scan computing devices and memory processing devices after power-on, and initialize and allocate address space for the computing devices and memory processing devices.

[0040] In one implementation, both the cache consistency root port and the cache consistency sub-port have a built-in cache consistency controller. The cache consistency controller is used to perform corresponding data processing at the physical layer, data link layer, and transaction layer, respectively. Specifically, the cache consistency controller is used to implement physical information exchange, interface initialization and maintenance at the physical layer, flow control and link state management at the data link layer, and packet encapsulation and parsing at the transaction layer.

[0041] In this embodiment, the cache consistency root port of the computing device is connected to the cache consistency sub-port of the memory processing device through a cache consistency protocol. This allows the computing device to synchronize the first result output from the query matrix and key-value matrix calculation stages of the model processing task to the memory processing device via the cache consistency protocol. Simultaneously, the memory processing device can synchronize the second result to the computing device via the cache consistency protocol. This achieves fast data transmission between the computing device and the memory processing device, saving memory space on the computing device and reducing the host processor overhead during the execution of the model processing task. To a certain extent, this improves the task processing efficiency of the hardware computing device.

[0042] Please see Figure 2 Another data processing system includes a host CPU, a GPU, and a CXL-PNM (CXL-Processing Near Memory). The host CPU, GPU, and CXL-PNM are connected via PCIe or a CXL bus. The GPU and PNM communicate via the CXL protocol.

[0043] (1) On the CPU side, the firmware is responsible for scanning the GPU and PNM devices and allocating a unified address space after power-on, and initializing registers such as the HDM Decoder in the GPU and PNM devices. The data path between the CPU and the two devices can be used for device configuration and task initialization, and can also be used for large-scale data transfer via PCIe DMA.

[0044] (2) The GPU can be implemented using the open-source Vortex architecture and is responsible for the main computational tasks (Part A) in the pre-filling and decoding stages of LLM inference. Data other than the KV Cache, including weights, activation functions, etc., are stored in on-chip memory such as VRAM. At the same time, to support CXL, the first CXL port RP (Root Port) needs to be integrated on the GPU.

[0045] (3) The CXL-PNM device is mainly responsible for implementing KV Cache storage, updating and access retrieval, as well as the attention mechanism calculation (Part B) of the decoding stage for frequently accessed KV Cache. At the same time, in order to support CXL, a second CXL port, CXL EP (Endpoint Port), needs to be integrated into the PNM device.

[0046] Please see Figure 3 The main implementations in GPUs include:

[0047] (1) The CXL controller, which implements the CXL protocol functions, has a three-layer structure. The Flex bus physical layer is responsible for physical information exchange, interface initialization and maintenance. The data link layer is responsible for reliable data transmission services, including link status and transaction management such as flow control and credit. The transaction layer handles transactions associated with each protocol, and performs packet encapsulation and parsing. In the GPU, the CXL controller is instantiated in CXL RP mode.

[0048] (2) The communication computing unit of the Vortex architecture GPU is a streaming multiprocessor (multiple SMs), connected to the AXI or ACE system bus through the last-level cache (LLC, L3 cache). Furthermore, the computing unit connects to the local memory controller and PCIe port (PCIe EP) via the AXI / ACE system bus to meet memory access requirements and communication needs with the host. After the host completes system scanning and enumeration, it configures the address decoder and related physical address range registers via PCIe bus transactions. The address decoder receives and decodes the address code sent by the computing unit, outputting enable signals for on-chip memory devices or extended video memory hardware devices.

[0049] (3) Since the latency of CXL extended video memory is relatively large compared to the GPU's local video memory, and considering that the CXL protocol transaction layer supports speculative read transactions, a data prefetching strategy can be designed in the memory request hardware unit, including a prefetch queue and a load queue buffer. If the load request address is included in the prefetch queue, the two requests are merged into one load request, and the request in the load queue has a higher priority than the prefetch request. The MUX arbitrator arbitrates the traffic from the two queues and sends the request to the CXL controller transaction layer. In addition, a store-to-load forwarding strategy is designed, meaning that if the address of a load or prefetch request is in the current store queue, the data can be retrieved directly from the store queue, and the loaded data result can be returned. Through the prefetching strategy and the store-to-load forwarding strategy, the access performance of extended video memory can be effectively improved.

[0050] Please see Figure 4 CXL-PNM can be implemented based on FPGA, and mainly includes:

[0051] (1) The CXL controller is instantiated as a CXL EP mode to implement the CXL protocol parsing and response functions on the device side. In addition, the PNM device also needs to instantiate an upstream port PCIe / CXL EP for connecting to the host. Figure 4 As shown in the upper left corner), communication with the host is realized. The host can use PCIe transactions to initialize and control the device, and scan and configure the configuration space registers (CSRs).

[0052] (2) The Key and Value caches in the LLM inference process are stored in the DDR of the NPM device. The PNM calculation unit implements a small number of calculation steps related to KV value access (QKV calculation) in the inference calculation. It is implemented by the PNM accelerated calculation unit on the device, which mainly includes: matrix calculation unit, vector calculation unit, buffer controller, buffer, DMA controller and other modules. The buffer stores load / store instructions and KV data required to execute calculation instructions. The DMA controller implements the control of batch data transfer between the buffer and the device DDR.

[0053] (3) The CXL.mem request processing unit receives the parsed CXL.mem request (from the GPU) from the CXL controller, accesses the HDM decoder register in the configuration space to translate it into a local address, and then sends a request to the memory controller (MC) to access DDR memory. The memory request arbitrator arbitrates the requests from the PNM computing unit and CXL.mem, and then forwards them to the MC memory controller for read and write data operations. The CXL.mem request is passed to the CXL controller through the Flex bus physical layer, arbitration, and multiplexer CXLARB / MUX.

[0054] If the model processing task is Transformer-based model inference computation, then the following example uses Transformer-based model inference computation. Given an input sequence, it first goes through a pre-filling stage, then through multiple decoding stages, generating output tokens one by one. The output token of each decoding stage serves as the input for the next decoding stage, and this process is repeated until a final token is generated.

[0055] Please see Figure 5 Each pre-padding or decoding stage follows a similar process: the input sequence is transformed into an embedding vector through an embedding layer, and then processed by N cascaded decoding layers to generate an output embedding vector, which is then converted into an output token. The decoding layer has two key sub-layers: self-attention and a feedforward network (FFN) layer. The self-attention layer consists of a QKV generation layer, an attention computation layer, and a fully connected layer representing the linear mapping of the attention layer's output. The attention computation mainly involves: d and k represent the embedding dimensions of matrix K.

[0056] It should be noted that each pre-filling or decoding stage is calculated according to the following process: the input sequence is converted into an embedding vector through the embedding layer, and then processed by N cascaded decoding layers to generate an output embedding vector, which is then converted into an output token. The decoding layer has two key sub-layers: Self-Attention and Feedforward Network (FFN) layers. The Self-Attention layer consists of a QKV generation layer, an attention calculation layer, and a fully connected layer (linear mapping layer output by the attention layer). Decoding stage Part A consists of an embedding layer and a decoding layer (FFN, QKV generation layer with self-attention calculation, and linear mapping layer output by the attention layer). The embedding layer maps discrete word IDs to a continuous vector space; the QKV generation layer generates query, key, and value vectors using linear mapping; the attention layer performs weighted combination of these vectors to complete information fusion; the FFN stage is a non-linear mapping used to further abstract feature representations.

[0057] Therefore, Figure 5 The hardware and data interaction processes required for each stage of the computation are shown below. Figure 6 As shown, it includes:

[0058] (1) The computation of the pre-filling stage is performed on the GPU. Its attention layer computation generates a key-value (KV) matrix for the subsequent decoding stage, which is the KV Cache. It is stored in the DDR memory of PNM through batch transfer via DMA. The size of the K matrix and the V matrix is ​​s0×d (s0 is the length of the input sequence and d is the embedding dimension).

[0059] (2) The input to the decoding stage is a single token generated in the previous stage, and the input to each decoder is also a vector. The attention calculation mainly consists of GEMV operations, using the q vector of the current input token and the aggregated KV matrix, calculated on the PNM computing device shown above. Specifically, regarding the aggregated KV matrix, for each new token decoded, corresponding q, k, and v vectors are generated. The k and v vectors are appended in real-time to the KV matrix passed down from the previous stage, forming a continuously growing aggregated KV matrix in the PNM memory, such as... Figure 5 The Attention mechanism is shown in the rightmost box. In this process, the q vector has a dimension of 1×d, and the K and V matrices have a dimension of s×d, where s represents the sum of s0 and the number of tokens generated in the current stage.

[0060] Accordingly, this embodiment has the following advantages: (1) The attention calculation is offloaded to the PNM. Unlike the original reading of the entire s×d dimension KV matrix, only the current 1×d dimension q, k, v matrix and output o matrix are transmitted on the card interconnect bus. The data dimension of the transmission is greatly reduced (originally the complete KV is transmitted, now a one-dimensional one is transmitted). This advantage is more obvious under the trend of long sequence inference. (2) Regarding the data transmission granularity, taking GPT-3 as an example, a 96-head multi-head attention mechanism is adopted. If the single-head embedding dimension is 128 and FP16 quantization is used, the size of the q / k / v / o matrix of each head is 256B; if the single-head embedding dimension is 64 and INT8 quantization is used, the size of the q / k / v / o matrix of each head is 64B. In the decoding stage, each decoder will frequently perform q, k, v vector writing and o vector reading operations, and the data transmission is relatively frequent. The CXL-based extended video memory solution implicitly and automatically synchronizes the updated QKV vectors to the extended video memory by leveraging the CXL bus hardware to maintain data consistency. This achieves efficient synchronization of fine-grained data (synchronization between the GPU cache and DDR in the PNM), which can significantly reduce transmission latency and software-level synchronization overhead. This allows data transmission to overlap with the computation stage efficiently, thereby shortening the overall inference time.

[0061] As can be seen, this embodiment implements KV Cache offloading for GPU graphics cards, which can significantly alleviate the storage pressure on KV Cache. Specifically, the KV Cache, which needs to be accessed frequently during the decoding stage, is offloaded to the GPU's extended video memory card (i.e., PNM), reducing the frequent transmission of KV matrices on the bus, improving communication efficiency, and effectively improving the overall performance of large model inference. CXL interconnect technology supports automatic implicit hardware updates of computational data. The KV tensors generated by each token in the GPU during the decoding stage are automatically updated to the extended video memory through the CXL bus in a hardware-maintained consistency manner, greatly reducing software overhead and allowing for efficient overlap with the computation stage, shortening the overall inference time.

[0062] The following describes a data processing method provided by an embodiment of the present invention. The data processing method described below can be referred to in conjunction with other embodiments described herein.

[0063] This invention discloses a data processing method applied to a computing device in a data processing system, comprising: executing a pre-filling stage and a query matrix and key-value matrix calculation stage of a model processing task sent by a host in the data processing system; synchronizing a first result output by the query matrix and key-value matrix calculation stage to a memory processing device in the data processing system via a cache consistency protocol, so that the memory processing device executes an attention calculation stage of the model processing task based on the first result; synchronizing a second result output by the attention calculation stage to the computing device via the cache consistency protocol; and determining the task result of the model processing task based on the second result.

[0064] The host connects the computing device and the memory processing device; the cache coherence root port of the computing device is connected to the cache coherence sub-port of the memory processing device through the cache coherence protocol.

[0065] In one embodiment, the computing device includes: a first computing core and a first memory; the first computing core is connected to the first memory and a cache consistency root port; the first computing core is used to: perform a pre-filling stage and a query matrix and key-value matrix calculation stage; the first memory is used to: store the input data of the model processing task, the output result of the pre-filling stage, and a first result.

[0066] In one embodiment, the memory processing device includes: a second computing core and a second memory; the second computing core is connected to the second memory and a cache coherence subport; the second computing core is used to: perform an attention calculation phase; the second memory is used to: store the first result and the second result.

[0067] In one embodiment, the second memory includes a memory controller and a storage medium; the memory controller is configured to: parse synchronization requests from a cache consistency subport and access the storage medium based on the parsing result; the storage medium is configured to: respond to the access from the memory controller.

[0068] In one embodiment, the storage medium is a double-rate synchronous dynamic random access memory.

[0069] In one embodiment, the computing device includes: a memory request processing unit; the memory request processing unit includes: a prefetch queue and a load queue; the prefetch queue is used to: store the addresses of each request in the load queue; the load queue is used to: store requests to be loaded.

[0070] In one embodiment, the memory request processing unit includes a storage queue; the storage queue is used to store the addresses of each request in the load queue and / or to store the addresses of each request in the prefetch queue.

[0071] In one embodiment, the second computing core includes: a matrix computing unit for performing matrix calculations included in the attention computing stage; a vector computing unit for performing vector calculations included in the attention computing stage; a buffer controller for managing a buffer; a direct memory access controller for enabling data interaction between the buffer and a second memory; and a buffer for storing computing commands included in the attention computing stage.

[0072] In one implementation, the host connects the computing device and the memory processing device via a high-speed serial computer expansion bus and / or a cache coherence protocol.

[0073] In one implementation, the host communicates with the computing device and the memory processing device respectively using direct memory access technology.

[0074] In one implementation, the host is used to: scan computing devices and memory processing devices after power-on, and initialize and allocate address space for the computing devices and memory processing devices.

[0075] In one implementation, both the cache consistency root port and the cache consistency sub-port have a built-in cache consistency controller; the cache consistency controller is used to perform corresponding data processing at the physical layer, data link layer and transaction layer respectively.

[0076] In one implementation, the cache consistency controller is used to: implement physical information exchange, interface initialization and maintenance at the physical layer, implement flow control and link state management at the data link layer, and implement data packet encapsulation and parsing at the transaction layer.

[0077] In one implementation, the computing device is used to: take the three 1×d-dimensional matrices output from the query matrix and key-value matrix calculation phase as the first result.

[0078] In one implementation, the memory processing device is used to: take the 1×d-dimensional fusion vector output from the attention calculation stage as a second result.

[0079] In one implementation, the computing device is used to: read a second result from a memory processing device using a prefetch strategy.

[0080] For more detailed information on the working process of each module and unit in this embodiment, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0081] As can be seen, this embodiment provides a data processing device that enables fast data transfer between computing devices and memory processing devices. This saves memory space on computing devices and reduces the overhead of the host processor during model processing tasks, thereby improving the task processing efficiency of hardware computing devices to a certain extent.

[0082] The following describes an electronic device provided by an embodiment of the present invention. The electronic device described herein can be referred to in conjunction with other embodiments described herein. The electronic device in this embodiment can be a host, a computing device, or a memory processing device, or any functional module of these devices.

[0083] This invention discloses an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the method disclosed in any of the above embodiments.

[0084] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: executing the pre-filling stage and the query matrix and key-value matrix calculation stage of the model processing task sent by the host in the data processing system; and synchronizing the first result output by the query matrix and key-value matrix calculation stage to the memory processing device in the data processing system through a cache consistency protocol.

[0085] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: based on the first result, it executes the attention calculation stage of the model processing task, and synchronizes the second result output by the attention calculation stage to the computing device through a cache consistency protocol.

[0086] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: determining the task result of the model processing task based on the second result.

[0087] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: performing a pre-filling stage and a query matrix and key-value matrix calculation stage.

[0088] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the input data of the model processing task, the output results of the pre-filling stage, and the first result.

[0089] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: performing an attention calculation stage.

[0090] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the first result and the second result.

[0091] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: parsing the synchronization request from the cache consistency sub-port and accessing the storage medium according to the parsing result.

[0092] In this embodiment, when the processor executes the computer program stored in the memory, it may specifically implement the following steps: responding to access by the memory controller.

[0093] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the addresses of each request in the loading queue.

[0094] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the request to be loaded.

[0095] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the addresses of each request in the load queue, and / or storing the addresses of each request in the prefetch queue.

[0096] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: implementing matrix calculations included in the attention calculation stage.

[0097] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: implementing vector calculations included in the attention calculation stage.

[0098] In this embodiment, when the processor executes the computer program stored in the memory, it may specifically implement the following steps: managing the buffer.

[0099] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: realize data interaction between the buffer and the second memory.

[0100] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: storing the calculation commands included in the attention calculation stage.

[0101] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: communicating with the computing device and the memory processing device respectively using direct memory access technology.

[0102] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: after power-on, scan the computing device and memory processing device, and initialize and allocate address space for the computing device and memory processing device.

[0103] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: performing corresponding data processing at the physical layer, data link layer, and transaction layer respectively.

[0104] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: physical information exchange, interface initialization and maintenance at the physical layer, flow control and link state management at the data link layer, and data packet encapsulation and parsing at the transaction layer.

[0105] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: taking the three 1×d-dimensional matrices output from the query matrix and key-value matrix calculation stages as the first result.

[0106] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: taking the 1×d-dimensional fusion vector output by the attention calculation stage as the second result.

[0107] In this embodiment, when the processor executes the computer program stored in the memory, it can specifically implement the following steps: reading the second result from the memory processing device using a prefetch strategy.

[0108] Furthermore, embodiments of the present invention also provide an electronic device. The aforementioned electronic device can be, for example,... Figure 7 The server shown can also be as follows: Figure 8 The terminal shown. Figure 7 and Figure 8 These are all structural diagrams of an electronic device according to an exemplary embodiment, and the content in the diagrams should not be considered as any limitation on the scope of the invention.

[0109] Figure 7 This is a schematic diagram of a server structure provided in an embodiment of the present invention. The server may specifically include: at least one processor, at least one memory, a power supply, a communication interface, an input / output interface, and a communication bus. The memory stores a computer program, which is loaded and executed by the processor to implement the relevant steps in the data processing disclosed in any of the foregoing embodiments.

[0110] In this embodiment, the power supply is used to provide operating voltage for each hardware device on the server; the communication interface can create a data transmission channel between the server and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this invention, and is not specifically limited here; the input / output interface is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0111] In addition, the memory, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored on it include operating system, computer programs and data, etc., and the storage method can be temporary storage or permanent storage.

[0112] The operating system manages and controls the various hardware devices and computer programs on the server to enable the processor to perform operations and processes on the data in the memory. It can be Windows Server, Netware, Unix, Linux, etc. In addition to computer programs capable of performing the data processing methods disclosed in any of the foregoing embodiments, the computer programs may further include computer programs capable of performing other specific tasks. The data may include application update information and application developer information.

[0113] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present invention. The terminal may include, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer.

[0114] Typically, the terminal in this embodiment includes a processor and a memory.

[0115] The processor may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor can be implemented using at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor may also include a main processor and coprocessors. The main processor, also known as the CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor may also include an AI (Artificial Intelligence) processor, which handles computational operations related to machine learning.

[0116] The memory may include one or more computer non-volatile storage media, which may be non-transitory. The memory may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory is used to store at least the following computer program, which, after being loaded and executed by the processor, is capable of implementing the relevant steps in the data processing method executed by the terminal side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory may also include operating systems and data, and the storage method may be temporary or permanent storage. The operating system may include Windows, Unix, Linux, etc. The data may include, but is not limited to, application update information.

[0117] In some embodiments, the terminal may further include a display screen, an input / output interface, a communication interface, a sensor, a power supply, and a communication bus.

[0118] Those skilled in the art will understand that Figure 8 The structure shown does not constitute a limitation on the terminal and may include more or fewer components than illustrated.

[0119] The following describes a non-volatile storage medium provided by an embodiment of the present invention. The non-volatile storage medium described below can be referred to in conjunction with other embodiments described herein.

[0120] A non-volatile storage medium is provided for storing a computer program, wherein the computer program, when executed by a processor, implements the data processing method disclosed in the foregoing embodiments. The non-volatile storage medium is a computer-readable non-volatile storage medium, which, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored thereon include an operating system, computer programs, and data, and the storage method can be temporary storage or permanent storage.

[0121] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: executing the pre-filling stage and the query matrix and key-value matrix calculation stage of the model processing task sent by the host in the data processing system; and synchronizing the first result output by the query matrix and key-value matrix calculation stage to the memory processing device in the data processing system through a cache consistency protocol.

[0122] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: based on the first result, execute the attention calculation stage of the model processing task, and synchronize the second result output by the attention calculation stage to the computing device through a cache consistency protocol.

[0123] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: determining the task result of the model processing task based on the second result.

[0124] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: performing a pre-filling stage and a query matrix and key-value matrix calculation stage.

[0125] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the input data of the model processing task, the results output in the pre-filling stage, and the first result.

[0126] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: performing an attention calculation stage.

[0127] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the first result and the second result.

[0128] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: parsing the synchronization request from the cache consistency sub-port and accessing the storage medium according to the parsing result.

[0129] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: responding to access by the memory controller.

[0130] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the addresses of each request in the loading queue.

[0131] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the request to be loaded.

[0132] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the addresses of each request in the load queue, and / or storing the addresses of each request in the prefetch queue.

[0133] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: implementing matrix calculations included in the attention calculation stage.

[0134] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: implementing vector calculations included in the attention calculation stage.

[0135] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: managing the buffer.

[0136] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: realize data interaction between the buffer and the second memory.

[0137] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: storing the calculation commands included in the attention calculation stage.

[0138] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: communicating with the computing device and the memory processing device respectively using direct memory access technology.

[0139] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: after power-on, scan the computing device and memory processing device, and initialize and allocate address space for the computing device and memory processing device.

[0140] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: performing corresponding data processing at the physical layer, data link layer, and transaction layer respectively.

[0141] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: physical information exchange, interface initialization and maintenance at the physical layer, flow control and link state management at the data link layer, and data packet encapsulation and parsing at the transaction layer.

[0142] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: taking the three 1×d-dimensional matrices output in the query matrix and key-value matrix calculation stages as the first result.

[0143] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: taking the 1×d-dimensional fusion vector output by the attention calculation stage as the second result.

[0144] In this embodiment, when the processor executes the computer program stored in the non-volatile storage medium, it can specifically implement the following steps: reading the second result from the memory processing device using a prefetch strategy.

[0145] The following describes a computer program product provided by an embodiment of the present invention. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0146] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed data processing method.

[0147] Embodiments of the present invention also provide another computer program product, including a non-volatile computer-readable storage medium for storing a computer program that, when executed by a processor, implements the steps in any of the above embodiments.

[0148] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0149] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of non-volatile storage medium known in the art.

[0150] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only intended to help understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data processing system, characterized in that, include: Mainframe, computing devices, and memory processing devices; The host is connected to the computing device and the memory processing device; The cache coherence root port of the computing device is connected to the cache coherence sub-port of the memory processing device through a cache coherence protocol. The host is used to: send model processing tasks to the computing device; The computing device is used to: perform the pre-filling stage and the query matrix and key-value matrix calculation stage of the model processing task, and synchronize the first result output by the query matrix and key-value matrix calculation stage to the memory processing device through a cache consistency protocol; The memory processing device is used to: perform the attention calculation stage of the model processing task based on the first result, and synchronize the second result output by the attention calculation stage to the computing device through a cache consistency protocol; The computing device is used to: determine the task result of the model processing task based on the second result.

2. The system according to claim 1, characterized in that, The computing device includes: a first computing core and a first memory; the first computing core is connected to the first memory and the cache coherence root port; The first computing core is used to: perform the pre-filling stage and the query matrix and key-value matrix calculation stage; The first memory is used to store the input data of the model processing task, the output result of the pre-filling stage, and the first result.

3. The system according to claim 1, characterized in that, The memory processing device includes: a second computing core and a second memory; the second computing core is connected to the second memory and the cache coherence subport; The second computational core is used to: perform the attention computation phase; The second memory is used to store the first result and the second result.

4. The system according to claim 3, characterized in that, The second memory includes: a memory controller and a storage medium; The memory controller is used to: parse synchronization requests from the cache consistency sub-port and access the storage medium based on the parsing results; The storage medium is used to respond to access by the memory controller.

5. The system according to claim 4, characterized in that, The storage medium is a double-rate synchronous dynamic random access memory.

6. The system according to claim 2, characterized in that, The computing device includes: a memory request processing unit; the memory request processing unit includes: a prefetch queue and a load queue; The prefetch queue is used to store the addresses of each request in the loading queue; The loading queue is used to store requests to be loaded.

7. The system according to claim 6, characterized in that, The memory request processing unit includes a storage queue; the storage queue is used to store the addresses of each request in the loading queue and / or to store the addresses of each request in the prefetch queue.

8. The system according to claim 3, characterized in that, The second computational core includes: The matrix calculation unit is used to: perform the matrix calculations included in the attention calculation stage; The vector computation unit is used to: perform the vector computations included in the attention computation stage; The buffer controller is used to: manage buffers; The direct memory access controller is used to: enable data interaction between the buffer and the second memory; The buffer is used to store the computation commands included in the attention computation phase.

9. The system according to claim 1, characterized in that, The host connects the computing device and the memory processing device via a high-speed serial computer expansion bus and / or cache coherence protocol.

10. The system according to claim 9, characterized in that, The host communicates with the computing device and the memory processing device respectively using direct memory access technology.

11. The system according to claim 1, characterized in that, The host is used to: scan the computing device and the memory processing device after power-on, and initialize and allocate address space for the computing device and the memory processing device.

12. The system according to claim 1, characterized in that, Both the cache consistency root port and the cache consistency sub-port have built-in cache consistency controllers. The cache consistency controller is used to perform corresponding data processing at the physical layer, data link layer, and transaction layer, respectively.

13. The system according to claim 12, characterized in that, The cache consistency controller is used to: implement physical information exchange, interface initialization and maintenance at the physical layer, implement flow control and link state management at the data link layer, and implement data packet encapsulation and parsing at the transaction layer.

14. The system according to any one of claims 1 to 13, characterized in that, The computing device is used to: take the three 1×d-dimensional matrices output from the query matrix and key-value matrix calculation stage as the first result.

15. The system according to any one of claims 1 to 13, characterized in that, The memory processing device is used to: take the 1×d-dimensional fusion vector output by the attention calculation stage as the second result.

16. The system according to any one of claims 1 to 13, characterized in that, The computing device is used to: read the second result from the memory processing device using a prefetch strategy.

17. A data processing method, characterized in that, Computing devices used in data processing systems include: The data processing system performs the pre-filling phase and the query matrix and key-value matrix calculation phase for the model processing task sent by the host. The first result output from the query matrix and key-value matrix calculation stage is synchronized to the memory processing device in the data processing system through a cache consistency protocol, so that the memory processing device can execute the attention calculation stage of the model processing task based on the first result, and the second result output from the attention calculation stage is synchronized to the computing device through the cache consistency protocol; The task result of the model processing task is determined based on the second result; The host connects the computing device and the memory processing device; the cache coherence root port of the computing device is connected to the cache coherence sub-port of the memory processing device through a cache coherence protocol.

18. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the method as described in claim 17.

19. A non-volatile storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the method as described in claim 17.

20. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method of claim 17.

Images