A data processing method and a data processing device, and a storage medium

Abstract

The application discloses a data processing method and device and a storage medium, and belongs to the technical field of computers, and the method comprises the following steps: acquiring a first dense data block set in a first matrix block, wherein the first matrix block belongs to a first matrix; acquiring a second dense data block set in a second matrix block, wherein the second matrix block belongs to a second matrix; and performing matrix multiplication operation on each dense data block in the first dense data block set and each dense data block in the second dense data block set respectively to obtain a first matrix calculation result. Thus, the matrix is divided into a plurality of matrix blocks, the matrix blocks are further divided into data blocks of a finer granularity, and the matrix multiplication operation is performed by taking the data block as a minimum calculation unit, so that the originally huge matrix multiplication operation is decomposed into a plurality of independent and small-scale data block matrix multiplication operations, the numerical stability of the whole calculation process is ensured, and the precision of a neural network model is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data processing method, a data processing apparatus, and a storage medium. Background Technology

[0002] With the development of neural networks, the demand for computing power is becoming increasingly strong, which has also spurred the development of many computing-intensive chips, such as tensor processing units (TPUs), graphics processing units (GPUs), and neural processing units (NPUs). These chips include computing units that can perform matrix multiplication operations. Furthermore, due to the data sparsity inherent in artificial intelligence (AI) scenarios, introducing sparsity characteristics into the chip's computing units can improve computing power and cost-effectiveness.

[0003] Currently, matrix multiplication is taken as A In the B=C example, matrix multiplication consists of multiple inner product calculation units, each of which performs the inner product operation on two vectors. For example, Ai is a dense vector in the i-th row of matrix A, and Bi is a sparse vector in the i-th column of matrix B, which has been compressed according to a 2:4 structured sparsity. By selecting the corresponding data in Ai based on the index data B_index in matrix B, the matrix multiplication can be performed.

[0004] Existing technologies use a fixed sparsity ratio, which reduces the accuracy of neural network models when combined with low-precision quantization techniques. Summary of the Invention

[0005] This application provides a data processing method, a data processing apparatus, and a storage medium for improving the accuracy of neural network models and reducing the storage resource consumption and access overhead of sparse information.

[0006] In a first aspect, this application provides a data processing method. This method can be applied to a processor, which includes a matrix computation unit. The method includes: acquiring a first dense data block set in a first matrix partition; wherein the first dense data set includes multiple dense data blocks, the first matrix partition belongs to a first matrix, and the first matrix includes multiple matrix partitions including the first matrix partition; acquiring a second dense data block set in a second matrix partition; wherein the second dense data set includes at least one dense data block, the second matrix partition belongs to a second matrix, and the second matrix includes multiple matrix partitions including the second matrix partition; and performing matrix multiplication operations between each dense data block in the first dense data block set and each dense data block in the second dense data block set to obtain a first matrix computation result.

[0007] The dense data block set includes only dense data blocks and excludes sparse data blocks.

[0008] Based on the above method, this application divides the matrix into multiple matrix blocks, further divides the matrix blocks into finer-grained data blocks, and performs matrix multiplication operations with data blocks as the smallest computing unit. This decomposes the originally large matrix multiplication operation into multiple independent, small-scale data block matrix multiplication operations, effectively avoiding the risk of numerical overflow caused by coarse-grained one-time accumulation calculations. At the same time, it reduces the error accumulation caused by multiple rounding operations, ensuring the numerical stability of the entire calculation process, thereby improving the accuracy of the final output result of the neural network model and contributing to the improvement of the accuracy of the neural network model.

[0009] Moreover, this application employs asymmetric sparsity, performing sparsity processing only on the first matrix block while maintaining the complete dense state of the second matrix block. This preserves fine-grained block computation, reduces the amount of data accessed per memory session, and avoids the additional index storage overhead and computational complexity caused by sparsity processing on the second matrix block.

[0010] Processors can be device-side processors, such as TPUs, GPUs, NPUs, etc. A processor can include at least one compute core. A compute core is a module within the processor used to implement computational capabilities. The compute cores of different types of processors can have different structures. For example, the compute core of an NPU can be an AI core, which includes matrix computation units and vector computation units designed for neural networks.

[0011] The processor includes any one of TPU, GPU, and NPU, and the controller in the processor includes an artificial intelligence central processing unit (AI CPU), the core of the AI ​​CPU, or a hardware module independent of the computing core in the processor.

[0012] In one possible implementation, the first matrix partition also includes sparse data blocks. Based on this approach, by filtering the first matrix partition, only the dense data block set used for computation is selected, skipping invalid computations of sparse data blocks. This computational mode can filter out sparse data blocks that do not need processing, reduce the number of invalid multiplication and accumulation operations in matrix multiplication, and lower computational complexity and resource consumption.

[0013] In one possible implementation, the method further includes: obtaining a third dense data block set for each of the remaining matrix blocks in the first matrix; obtaining a fourth dense data block set for each of the remaining matrix blocks in the second matrix; and performing matrix multiplication operations on each dense data block in the third dense data block set and each dense data block in the fourth dense data block set to obtain the matrix calculation result corresponding to each of the remaining matrix blocks.

[0014] Based on the above method, each remaining matrix in the first matrix is ​​divided into blocks, and its corresponding third dense data block set is obtained. The fourth dense data block set of the corresponding matrix block in the second matrix is ​​also obtained. Each dense data block in the third dense data block set is multiplied by each dense data block in the fourth dense data block set to obtain the corresponding matrix calculation result. All matrix calculation results are concatenated to obtain the complete output matrix.

[0015] In one possible implementation, obtaining a first dense data block set in a first matrix block includes: obtaining the first dense dataset from the first matrix block in column order based on a first sparse indication information; correspondingly, obtaining a second dense data block set in a second matrix block includes: obtaining the second dense dataset from the second matrix block in row order.

[0016] Based on the above approach, this application uses sparse indicator information at the data block granularity to indicate the sparse state of data blocks, which is less than the index storage required for 2:4 structured sparse data, thus reducing the storage resource consumption and memory access overhead of sparse information. In the form of sparse memory access, memory access is performed at the data block granularity according to the first sparse indicator information, and the first dense dataset is obtained from the first matrix block in column order, skipping sparse data blocks.

[0017] Let matrix multiplication be A In the B=C example, each matrix calculation unit is responsible for the matrix calculation of the first matrix block of A and the second matrix block of B. Each matrix block is further divided into multiple fixed data blocks, for example, the matrix block granularity is 256. 256, while the data block granularity is 16. 16 / 32 32 / 64 Smaller granularity, such as 64. Because the first matrix is ​​divided into sparse blocks, one part consists of sparse data blocks and the other part consists of dense data blocks.

[0018] Since the second matrix block is in dense mode, the entire matrix block needs to be read. Therefore, the second dense data block set is obtained from the second matrix block in row order.

[0019] In one possible implementation, obtaining the first dense data block set in the first matrix block includes: determining the base address of multiple dense data blocks in the first dense dataset according to the first sparse indication information; and obtaining the first dense dataset in the first matrix block based on the base address of the multiple dense data blocks in the first dense dataset.

[0020] Based on the aforementioned method, the first sparse indicator information only needs to store the base address and optional block length of each dense data block, without needing to store the coordinates of each non-zero element. For example, when the block size is 16 columns, a 1024-column matrix block only requires 6 bits to represent a block number, and the storage overhead of the indicator information is 1-2 orders of magnitude lower than that of the element-level sparse format. At the same time, base address addressing is the easiest operation to implement in hardware, eliminating the need for complex index calculations, hash lookups, or out-of-order execution logic, thus reducing the complexity of the processing element (PE).

[0021] In one possible implementation, obtaining the first dense data block set in the first matrix block includes: obtaining all data blocks in the first matrix block; and obtaining the first dense dataset from all data blocks in column order based on a first sparse indication information.

[0022] Based on the aforementioned method, all data blocks in the first matrix block and the second dense data block set in the second matrix block are obtained in the form of dense memory access, wherein all data blocks include the first dense data block set, which includes multiple dense data blocks.

[0023] Since the first matrix is ​​divided into sparse blocks, consisting of sparse and dense data blocks, the first dense dataset is retrieved from all data blocks in column order based on the first sparsity indication information, skipping the sparse data blocks.

[0024] In one possible implementation, the first sparsity indication information is a sparse masking matrix, in which the values ​​of multiple elements are used to indicate the sparse state of the first matrix block.

[0025] For example, taking a sparse mask matrix as an example, bit-0 can be used to indicate that the data block at the corresponding position of the first matrix block is a sparse data block, and bit-1 indicates that the data block at the corresponding position of the first matrix block is a dense data block.

[0026] As the quantization bit width decreases, the storage overhead of index data used to represent the positions of non-zero elements increases significantly. Based on the aforementioned method, sparse indicator information at the data block granularity is used to indicate the sparse state of the data block, which is less than the index storage required for 2:4 structured sparseness, thus reducing the storage resource consumption and memory access overhead of sparse information.

[0027] In one possible implementation, each dense data block in the first dense data block set is multiplied by each dense data block in the second dense data block set to obtain a first matrix calculation result. This includes: performing matrix multiplication on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain a second matrix calculation result corresponding to the Nth column; wherein N, M, and i are positive integers.

[0028] Based on the aforementioned method, when performing matrix multiplication, the first matrix block is divided into columns and the second matrix block is divided into rows to perform matrix calculations at the data block granularity level, and the calculation is performed by using the outer product method.

[0029] It should be noted that by using the outer product method for calculation, each column of matrix A and each row of matrix B only need to be read from global memory once, and can be reused by the PE cells of the entire systolic array, which can reduce memory bandwidth requirements.

[0030] In one possible implementation, i equals 1; or, If i is greater than or equal to 2, the i dense data blocks in the Nth column of the first dense data block set are stored in the register.

[0031] Based on the aforementioned method, when there are more than two dense data blocks in the Nth column of the first matrix block tileA, each dense data block corresponds to all data blocks in the corresponding row direction of the second matrix block tileB. Therefore, by storing i dense data blocks in a double buffer register, each time a data block of the second matrix block tileB is read in, it is calculated together with the i dense data blocks stored in tileA. This is equivalent to reducing the data access of tileB by half, thus reducing the access overhead.

[0032] Meanwhile, when processing the last block of tile B, the next dense data block of the first matrix segment tile A can be preloaded into the register double buffer, thereby hiding the data loading delay of the next round of calculation and further reducing the overall memory access overhead.

[0033] In one possible implementation, the method also includes: If the Nth column of the first matrix block does not include dense data blocks, skip the following steps: perform matrix multiplication based on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set, which can further reduce the overall memory access overhead.

[0034] In one possible implementation, i is greater than or equal to 2, and the method further includes: before obtaining the second matrix calculation result corresponding to the Nth column, obtaining j dense data blocks in the (N+1)th column of the first dense data block set, and obtaining each dense data block in the Mth row of the second dense data block set, where j is a positive integer; after obtaining the second matrix calculation result corresponding to the Nth column, performing matrix multiplication operations based on the j dense data blocks in the (N+1)th column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the third matrix calculation result corresponding to the (N+1)th column of the first dense data block set.

[0035] Based on the aforementioned method, when performing matrix multiplication, the first matrix block is divided into columns and the second matrix block is divided into rows, and the matrix calculation is performed at the data block granularity, using the outer product method.

[0036] It should be noted that by using the outer product method for calculation, each column of matrix A and each row of matrix B only need to be read from global memory once, and can be reused by the PE cells of the entire systolic array, which can reduce memory bandwidth requirements.

[0037] Moreover, by overlapping the data loading process of column N+1 with the calculation process of column N, the memory access overhead during column switching can be reduced, ensuring the continuous operation of the computing unit.

[0038] In one possible implementation, j equals 1; or, If j is greater than or equal to 2, the j dense data blocks in the (N+1)th column of the first dense data block set are stored in the register.

[0039] Based on the aforementioned method, when j is greater than or equal to 2, the j dense data blocks can be stored in a register double buffer. Each time a data block of the second matrix segment tileB is read, it can be calculated simultaneously with the stored j dense data blocks, thereby reducing the number of data accesses of the second matrix segment tileB to 1 / j of the original, further reducing the number of repeated data moves of the second matrix segment, and improving the overall efficiency of matrix multiplication operations and memory access bandwidth utilization.

[0040] In one possible implementation, the method further includes: obtaining a fifth dense data block set from a third matrix block in row order based on second sparse indication information, the fifth dense data block set including multiple dense data blocks; the third matrix block has a different partitioning dimension than the first matrix block; obtaining a sixth dense data block set from a fourth matrix block in column order based on second sparse indication information, the sixth dense data block set including at least one dense data block; the fourth matrix block has a different partitioning dimension than the second matrix block, and the values ​​of multiple elements in the second sparse indication information are used to indicate the sparsity state of the third matrix calculation result obtained by performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set; and performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set to obtain the third matrix calculation result.

[0041] Based on the aforementioned method, by adjusting the segmentation dimension and data acquisition order of the matrix blocks according to the data flow characteristics of low-precision FlashAttention scenarios, it is possible to deeply integrate with the block calculation logic of FlashAttention, thereby improving the overall efficiency of matrix multiplication operations in this scenario.

[0042] Secondly, this application provides a data processing apparatus. The data processing apparatus can be applied to a processor.

[0043] The data processing device includes: an acquisition unit and a matrix calculation unit; wherein: The acquisition unit is used to acquire a first dense data block set in a first matrix block; wherein, the first dense data set includes multiple dense data blocks, the first matrix block belongs to a first matrix, and the first matrix includes multiple matrix blocks including the first matrix block; The acquisition unit is further configured to acquire a second dense data block set in the second matrix block; wherein the second dense data set includes at least one dense data block, the second matrix block belongs to a second matrix, and the second matrix includes multiple matrix blocks including the second matrix block; The matrix calculation unit is used to perform matrix multiplication operations on each dense data block in the first dense data block set and each dense data block in the second dense data block set to obtain the first matrix calculation result.

[0044] In one possible implementation, the first matrix block also includes sparse data blocks.

[0045] In one possible implementation, the acquisition unit is further configured to acquire a third dense data block set for each of the remaining matrix blocks in the first matrix; Obtain the set of the fourth dense data blocks for each of the remaining matrix blocks in the second matrix; The matrix calculation unit is further configured to perform matrix multiplication operations on each dense data block in the third dense data block set and each dense data block in the fourth dense data block set, so as to obtain the matrix calculation result corresponding to each of the remaining matrix blocks.

[0046] In one possible implementation, the acquisition unit is specifically used to acquire the first dense dataset from the first matrix in column order based on the first sparse indication information. Accordingly, obtaining the second dense data block set in the second matrix partition includes: The second dense dataset is obtained from the second matrix blocks in row order.

[0047] In one possible implementation, the acquisition unit is specifically used to determine the base address of multiple dense data blocks in the first dense dataset based on the first sparse indication information. Based on the base addresses of multiple dense data blocks in the first dense dataset, the first dense dataset in the first matrix block is obtained.

[0048] In one possible implementation, the first sparsity indication information is a sparse masking matrix, and the values ​​of multiple elements in the sparse masking matrix are used to indicate the sparse state of the first matrix block.

[0049] In one possible implementation, the matrix calculation unit is specifically used to perform matrix multiplication operations based on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set, so as to obtain the second matrix calculation result corresponding to the Nth column. Wherein, N, M, and i are positive integers.

[0050] In one possible implementation, the data processing device further includes: a storage unit; The i equals 1; or, The i is greater than or equal to 2, and the storage unit is used to store the i dense data blocks in the Nth column of the first dense data block set in the register.

[0051] In one possible implementation, if the Nth column of the first matrix block does not include a dense data block, the following step is skipped: performing matrix multiplication based on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set.

[0052] In one possible implementation, i is greater than or equal to 2; Before obtaining the calculation result of the second matrix corresponding to the Nth column, the acquisition unit is specifically used to acquire j dense data blocks in the N+1th column of the first dense data block set, and to acquire each dense data block in the Mth row of the second dense data block set, where j is a positive integer; After obtaining the second matrix calculation result corresponding to the Nth column, the matrix calculation unit is further used to perform matrix multiplication operations on the j dense data blocks in the N+1th column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the third matrix calculation result corresponding to the N+1th column of the first dense data block set.

[0053] In one possible implementation, j equals 1; or, The storage unit, where j is greater than or equal to 2, is used to store the j dense data blocks in the (N+1)th column of the first dense data block set in a register.

[0054] In one possible implementation, the acquisition unit is further configured to acquire a fifth dense data block set from the third matrix block in row order, the fifth dense data set comprising multiple dense data blocks; the third matrix block has a different partitioning dimension than the first matrix block. Based on the second sparse indication information, the sixth dense data block set is obtained from the fourth matrix block in column order. The sixth dense data set includes at least one dense data block. The fourth matrix block has a different partitioning dimension than the second matrix block. The values ​​of multiple elements in the second sparse indication information are used to indicate the sparsity state of the third matrix calculation result obtained by performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set. The matrix calculation unit is also used to perform matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set to obtain the third matrix calculation result.

[0055] Thirdly, this application provides a processor. The processor is connected to a host in a communication system. The processor can be a device-side processor, including but not limited to an NPU or a general-purpose computing on graphics processing unit (GPU). The processor includes a computing core. Optionally, the processor also includes memory, also known as device memory or video memory. The processor is used to execute computer-readable instructions loaded into the memory to perform the data processing method as described in the first aspect of this application or any implementation thereof.

[0056] Fourthly, this application provides a communication system. The communication system includes a host and processors, for example, a host and multiple processors, the processors being configured to execute computer-readable instructions to perform the data processing method as described in the first aspect or any implementation thereof.

[0057] Fifthly, this application provides a computer-readable storage medium storing instructions that instruct a computing device or a cluster of computing devices to perform the data processing method described in the first aspect or any implementation thereof.

[0058] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, causes the computing device or cluster of computing devices to perform the data processing method described in the first aspect or any implementation thereof.

[0059] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods.

[0060] The technical effects of any of the design methods in aspects two through six can be found in the technical effects of the different design methods in aspect one above, and will not be repeated here. Attached Figure Description

[0061] To more clearly illustrate the technical methods of this application, the accompanying drawings used will be briefly described below.

[0062] Figure 1 A schematic diagram of a data center architecture provided for this application; Figure 2a and Figure 2b Schematic diagrams of some data processing platforms provided for this application; Figure 3 A schematic diagram of the data processing flow of a matrix calculation unit provided in this application; Figure 4A schematic diagram of a conventional matrix calculation provided in this application; Figure 5 A flowchart illustrating a data processing method provided in this application; Figure 6 This application provides an architectural diagram of a memory access method; Figure 7 A schematic diagram of a matrix block computation provided in this application; Figure 8 A schematic diagram of a matrix block memory access method provided in this application; Figure 9 A schematic diagram of an alternative memory access method provided in this application; Figure 10 A schematic diagram illustrating another matrix block computation method provided in this application; Figure 11 A schematic diagram of a matrix multiplication operation provided in this application; Figure 12 A schematic diagram illustrating another matrix multiplication operation provided in this application; Figure 13 A schematic diagram illustrating another matrix multiplication operation provided in this application; Figure 14 A schematic diagram illustrating another matrix multiplication operation provided in this application; Figure 15 A schematic diagram illustrating another matrix block computation method provided in this application; Figure 16 A schematic diagram of matrix partitioning provided in this application; Figure 17 A schematic diagram of the architecture of a computing device is provided for this application; Figure 18 A schematic diagram of the architecture of a computing device cluster provided in this application; Figure 19 This is a schematic diagram of another computing device cluster architecture provided in this application. Detailed Implementation

[0063] This application provides a data processing method, a data processing apparatus, and a storage medium for improving the accuracy of neural network models and reducing the storage resource consumption and access overhead of sparse information.

[0064] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.

[0065] As will be known to those skilled in the art, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0066] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. The terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the description of embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0067] First, let me explain the terminology that may be involved in the embodiments of this application: (1) Infrastructure The infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. Communication with the outside world is achieved through sensors; computing power is provided by intelligent chips, such as hardware acceleration chips (CPUs, NPUs, GPUs, ASICs, or FPGAs); the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0068] (2) Data

[0069] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0070] (3) Data processing

[0071] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0072] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0073] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0074] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0075] (4) General ability

[0076] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0077] (5) Smart products and industry applications

[0078] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0079] The following provides several typical application scenarios for the data processing method provided in the embodiments of this application: 1. Real-time inference in cloud-based recommendation systems: feature aggregation matrix multiplication after high-dimensional embedding layer lookup, batch matrix multiplication of multilayer perceptron, and 3D pulsating array NPU optimized for recommendation systems can efficiently process batch matrix multiplication of high-dimensional sparse features, reducing the server deployment cost of recommendation systems.

[0080] 2. Training and Inference of Large Language Models in the Cloud: The natural unstructured sparsity of the activation side, combined with the sparsity of weights, can achieve several times the computing power, reduce the training cost of large parameter models, improve the throughput of batch inference, and support real-time inference with longer context windows.

[0081] 3. Edge inference for computer vision tasks: Enable CV models with larger parameters (such as real-time semantic segmentation and 3D object detection) to run on low-power edge devices to meet the real-time requirements of autonomous driving and intelligent security.

[0082] 4. Training and inference of multimodal large models: The cloud-based 3D pulsating array NPU supports multimodal computing and can process sparse data of different dimensions simultaneously, reducing the training and inference costs of multimodal large models and supporting high-complexity tasks such as real-time image and text generation and video understanding.

[0083] It should be noted that the above application scenarios are for illustrative purposes only and are not intended to limit the scope of this application.

[0084] The data processing method provided in this application can be executed on a communication device, such as a server, or on a terminal device. The terminal device can be a mobile phone with text, voice, and image processing capabilities, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smartwatch, a wearable device (WD), or an autonomous vehicle, etc. This application does not limit the scope of the application to this method.

[0085] It is understood that the above example is only an exemplary introduction to one application scenario, and the data processing method of this application embodiment can be applied to multiple fields, and is not limited here.

[0086] The data processing method provided in this application embodiment can be applied to a computing device cluster, which may include one or more computing devices.

[0087] The type of computing device is not limited here. For example, any computing device can be a terminal device, a cloud server, a container, or a virtual machine, etc.

[0088] The computing device cluster can provide a data processing platform to implement the data processing method of the embodiments of this application. Specifically, a data processing platform can be provided in the terminal device and in the cloud server. Based on the respective data processing platforms of the terminal device or the cloud server, consistency and computing accuracy between the terminal side and the cloud side can be achieved, while also making full use of the differences in computing power between different devices. The specific form of the computing device cluster and the corresponding data processing platform is not limited here.

[0089] In one example, the cluster of computing devices can be used to implement a cloud management platform; in other words, the data processing platform of this application embodiment can be implemented through a cloud management platform.

[0090] A cloud management platform is used to manage the infrastructure that provides cloud services. It can provide computing, networking, and storage capabilities based on hardware and software resources. For example, a cloud management platform may include one or more data centers to provide cloud resources through one or more data centers.

[0091] The following is combined with Figure 1 The diagram shown illustrates an architecture for a data center.

[0092] Figure 1 In this data center, the cloud management platform communicates with one or more servers (such as...) via the data center's internal network. Figure 1 Servers 1 and 2 interact with each other. A server consists of a hardware layer and a software layer. The hardware layer includes the server's hardware configuration, such as peripheral component interconnect (PCI) devices (e.g., network cards, graphics processing units, GPUs, offloading cards, etc.) that can be plugged into PCI or PCIe slots. The software layer includes the operating system installed and running on the server (the operating system relative to a virtual machine can be called the host operating system). The host operating system contains a virtual machine manager (also called a hypervisor), whose role is to implement computational virtualization, network virtualization, and storage virtualization of virtual machines, and to manage virtual machines. A virtual machine (virtual machine) refers to a complete computer system simulated by software, possessing full hardware system functionality, and running in a completely isolated environment. Figure 1In the system architecture shown, a data center contains multiple servers, which can be used to run virtual machines. The specifications of the virtual machines can be the same or different. These virtual machines can also be called cloud servers (elastic compute service, ECS), elastic instances, etc., and different cloud service providers may use different names.

[0093] In one example of the embodiments of this application, the cloud management platform can be a public cloud platform. In this case, cloud service providers such as individuals or software developers with cloud resource development capabilities can provide cloud services to users. Users obtain cloud services through the Internet, but do not own cloud computing resources.

[0094] Specifically, in Figure 1 In the example shown, the cloud management platform can provide access interfaces (such as user interfaces or application programming interfaces, APIs). Users of the cloud management platform and cloud service providers can operate the client to remotely access the access interface to register cloud accounts and passwords on the cloud management platform. After the cloud management platform successfully authenticates the cloud accounts and passwords, they can log in to the cloud management platform to create, manage, log in to, and operate virtual machines in the cloud data center, thereby performing corresponding tasks through the cloud resources of the cloud management platform.

[0095] For example, such as Figure 2a In the example shown, the data processing platform that implements the data processing method in the embodiments of this application can be provided to users in the form of cloud services.

[0096] When data processing tasks require cross-device storage of graph data, some enterprises, organizations, or individuals can purchase cloud services that include the data processing platform. They can then use the cloud resources of the cloud management platform to perform the relevant data processing tasks and obtain the processing results of the input data from the cloud management platform.

[0097] This data processing platform can be implemented based on cloud resources such as one or more cloud data centers in the cloud management platform.

[0098] This data processing platform can be provided to users as an independent cloud service, and it is also compatible with computing frameworks such as TensorFlow and PyTorch. In other words, it can be used as an operator in computing frameworks such as TensorFlow and PyTorch, providing functions such as preprocessing, computation, feature extraction and / or other data processing for real-time and offline data.

[0099] Of course, the cloud management platform can also be other types of cloud management platforms, and this application embodiment does not limit this.

[0100] In another example, such as Figure 2b As shown, the data processing platform can be provided to the user in the form of a software product or a functional service within a software product. The software product can be deployed on the user's terminal device or on the enterprise user's server cluster. The user can purchase a software license to obtain the right to use the software product and, within the license period, implement the point data processing method of the embodiments of this application through the software product.

[0101] This application also provides a computing card. For example, the computing card can be an NPU, GPU, or TPU. The computing card includes computing cores and memory. A computing core is a module in the computing card used to implement computing capabilities. The computing cores of different types of computing cards can have different structures. For example, the computing core of an NPU can be an AI core, which includes matrix calculation units designed for neural networks; the computing core of a GPU includes multiple stream processors used for parallel processing of data computation tasks. The memory of the computing card is also called device memory or video memory, and typically includes global memory. The computing core is used to execute computer-readable instructions loaded into the memory to perform the data processing method provided in the embodiments of this application.

[0102] Figure 3 A schematic diagram of the data processing flow of a matrix computation unit is shown. The data processing flow of the matrix computation unit is mainly divided into five basic stages: data copyin unit, data splitting unit, data computation unit, data aggregation unit, and data copyout unit. Among them, CO1 is the first-level cache for computation output, CO2 is the second-level cache for computation output, VECIN is the vector input cache, VECOUT is the vector output cache, and BufPool1~BufPool4 are buffer pool 1 to buffer pool 4.

[0103] The data loading unit is responsible for loading operations, loading the data to be processed from global memory (GM) into the local memory of the matrix calculation unit; the data segmentation unit is used to segment the matrix; the data calculation unit is responsible for vector instruction calculation operations, completing specific vector operations; the data aggregation unit is responsible for data aggregation operations, aggregating the calculated data; and the data loading unit is responsible for data loading operations, outputting the aggregated data to memory.

[0104] It is understood that the above example is only an exemplary introduction to one application scenario, and the data processing method of this application embodiment can be applied to multiple fields, and is not limited here.

[0105] As introduced in the background section, the existing 2:4 structured sparsity technique with a fixed sparsity rate is prone to causing a significant deterioration in the accuracy of neural network models when combined with conventional low-precision quantization techniques such as 4-bit and 8-bit. Therefore, a separate low-precision quantization scheme is usually preferred.

[0106] Furthermore, when 2:4 structured sparsity is combined with low-precision quantization, the storage overhead of index data used to characterize the positions of non-zero elements increases significantly with the reduction of quantization bit width, requiring more chip storage and processing resources.

[0107] Combination Figure 4 The diagram shown illustrates an existing matrix computation, in which the multiplexer (MUX) is the core hardware component of the sparse matrix multiplication computation unit, used to implement sparse matrix data filtering and selection.

[0108] Taking a 4-bit quantization scenario as an example, the bit width of a single element in a sparse matrix is ​​4 bits, and every 4 consecutive elements require 4 bits of index data, which translates to an index overhead of 1 bit per element, significantly increasing the relative proportion. To reduce this overhead, existing technologies typically reuse 8-bit sparse processing logic and achieve a 2:4 structured sparse effect through a 4:8 structured sparse with positional constraints. This approach requires that the 4 non-zero elements selected from the 8 consecutive elements must be arranged in pairs, which further exacerbates the degradation of model accuracy.

[0109] To address the technical problems of low accuracy and high storage overhead of index data in existing neural network models, this application provides a data processing method, which will be described in detail below with reference to the accompanying drawings.

[0110] Please see Figure 5 This is a flowchart illustrating a data processing method provided in this application. The method can be applied to a processor, which includes a matrix calculation unit.

[0111] Optionally, the processor may include a device-side processor, such as a tensor processing unit (TPU), a graphics processing unit (GPU), or a neural processing unit (NPU). The processor may include at least one compute core. A compute core is a module within the processor used to implement computational capabilities. The compute cores of different types of processors may differ in structure. For example, the compute core of an NPU may be an AI core, which includes matrix computation units and vector computation units designed for neural networks.

[0112] The processor includes any one of a tensor processing unit (TPU), a graphics processing unit (GPU), or a neural processing unit (NPU), and the controller in the processor includes an artificial intelligence central processing unit (AI CPU), the core of the AI ​​CPU, or a hardware module independent of the computing core in the processor.

[0113] Specifically, the method includes: S501: Obtain the first dense data block set in the first matrix block; wherein, the first dense data set includes multiple dense data blocks, the first matrix block belongs to the first matrix, and the first matrix includes multiple matrix blocks including the first matrix block.

[0114] The dense data block set includes only dense data blocks and excludes sparse data blocks.

[0115] For A For matrix calculations of B=C, each matrix calculation unit is responsible for the matrix calculation of one matrix block (first matrix block) of matrix A and one matrix block (second matrix block) of matrix B. This application further divides each matrix block (tile) into multiple data blocks. For example, the granularity of a tile is 256. 256, while the granularity of a block can be 16. 16, 32 32, 64 Smaller particle sizes, such as 64.

[0116] It should be understood that the first matrix to be calculated comprises multiple first matrix blocks, and this application further divides each first matrix block into multiple data blocks. Taking a sparse matrix block as an example, the multiple data blocks may include multiple sparse data blocks and multiple dense data blocks.

[0117] This application divides a matrix into multiple matrix blocks, further subdivides these matrix blocks into finer-grained data blocks, and performs matrix multiplication operations using these data blocks as the smallest computational unit. This decomposes the originally massive matrix multiplication operation into multiple independent, small-scale data block matrix multiplication operations, effectively avoiding the risk of numerical overflow caused by coarse-grained one-time accumulation calculations. At the same time, it reduces the accumulation of errors caused by multiple rounding operations, ensuring the numerical stability of the entire calculation process. This improves the accuracy of the final output result of the neural network model and is beneficial to improving the precision of the neural network model.

[0118] In one possible implementation, the first sparse indication information can be a sparse mask matrix, in which the values ​​of multiple elements are used to indicate the sparse state of the first matrix block.

[0119] For example, taking a sparse mask matrix as an example, bit-0 in the matrix can be used to indicate that the data block at the corresponding position of the first matrix block is a sparse data block, and bit-1 indicates that the data block at the corresponding position of the first matrix block is a dense data block.

[0120] It should be noted that this application uses sparse indication information at the data block granularity to indicate the sparse state of the data block, which is less than the index storage required for 2:4 structured sparse, thus reducing the storage resource occupation and memory access overhead of sparse information.

[0121] In one possible implementation, the first sparse indicator information corresponding to the first matrix block can be obtained through offline preprocessing. The sparse mask matrix can be stored in a compressed format, that is, only the densely packed data blocks are stored, or it can be stored in an uncompressed format, directly retaining the original data of the matrix and only recording the corresponding sparse information.

[0122] S502: Obtain the second dense data block set in the second matrix block; wherein, the second dense data set includes at least one dense data block, the second matrix block belongs to the second matrix, and the second matrix includes multiple matrix blocks including the second matrix block.

[0123] Among them, for A For matrix calculation of B=C, each matrix calculation unit is responsible for the matrix calculation of one matrix block (first matrix block) of matrix A and one matrix block (second matrix block) of B. In this application, each matrix block (tile) is further divided into multiple data blocks (blocks).

[0124] It should be understood that the second matrix to be calculated comprises multiple second matrix blocks, and this application further divides each first matrix block into multiple data blocks.

[0125] S503: Perform matrix multiplication operations between each dense data block in the first dense data block set and each dense data block in the second dense data block set to obtain the first matrix calculation result.

[0126] In this step, the first matrix calculation result is obtained by performing matrix multiplication operations between each dense data block in the first dense data block set and each dense data block in the second dense data block set.

[0127] It should be understood that this application selects only the dense data block set used for computation by filtering the first matrix into blocks, skipping the invalid computation of sparse data blocks. This computation mode can filter out sparse data blocks that do not need to be processed, reduce the number of invalid multiplication and accumulation operations in matrix multiplication, and reduce computational complexity and computational resource consumption.

[0128] In one possible implementation, this application further obtains a third dense data block set for each of the remaining matrix blocks in the first matrix; obtains a fourth dense data block set for each of the remaining matrix blocks in the second matrix; and performs matrix multiplication operations between each dense data block in the third dense data block set and each dense data block in the fourth dense data block set to obtain the matrix calculation result corresponding to each of the remaining matrix blocks.

[0129] Understandably, this application obtains a third dense data block set corresponding to each of the remaining matrices in the first matrix, and obtains a fourth dense data block set corresponding to the matrix blocks in the second matrix; performs matrix multiplication operations on each dense data block in the third dense data block set and each dense data block in the fourth dense data block set to obtain the corresponding matrix calculation result; and concatenates all matrix calculation results to obtain a complete output matrix.

[0130] In this application, the remaining matrix blocks in the first matrix refer to all matrix blocks in the first matrix other than the processed first matrix blocks; the corresponding matrix blocks in the second matrix refer to the second matrix blocks that are paired with the remaining blocks of the first matrix under the matrix multiplication operation rules.

[0131] It is understandable that the division method and filtering rules of the third and fourth dense data block sets are consistent with those of the first and second dense data block sets mentioned above.

[0132] In one possible implementation, matrix multiplication is performed on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the second matrix calculation result corresponding to the Nth column; wherein, N, M and i are positive integers.

[0133] Specifically, matrix multiplication is performed on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the calculation result of the second matrix corresponding to the Nth column, until the block calculation of the first matrix is ​​completed.

[0134] In this embodiment, when performing matrix multiplication, the first matrix block performs matrix calculations in the column direction and the second matrix block performs matrix calculations in the row direction, using the outer product method.

[0135] It should be noted that by using the outer product method for calculation, each column of matrix A and each row of matrix B only need to be read from global memory once, and can be reused by all PEs of the entire pulsating array, which can reduce memory bandwidth requirements.

[0136] This application can obtain multiple dense data blocks through different memory access methods, which will be described below.

[0137] See Figure 6 The diagram illustrates an architecture for a memory access method. This method is primarily used in matrix multiplication (Matmul) operators, targeting a one-sided sparse input data pattern. L2 is a level 2 cache used to cache matrix tile data moved from high bandwidth memory (HBM). `buf` is a local buffer used to temporarily store matrix tile data read from the L2 cache for real-time access by the computation unit `CUBE`. L0 is a level 0 register file used to store intermediate multiplication and accumulation results and the final output data during the computation process.

[0138] First, the base address of the tile block corresponding to the weight matrix accessed by each computing core is pre-calculated and stored in a linked list. Since the weight matrix is ​​also stored in HBM in units of tiles, the linked list only needs to record the starting tile block number to be accessed by each computing core and the number of non-zero blocks within the corresponding tile block. The memory access engine of each computing core calculates the base address of the corresponding tile block in HBM based on the starting tile block number stored in the linked list, and determines the amount of memory access data based on the number of non-zero blocks within the tile block, completing the first stage of data transfer, that is, moving the required data from HBM to the chip's internal cache.

[0139] Subsequently, the second stage of data transfer is performed, which involves reading data from the chip's internal cache and sending it to the matrix calculation unit for computation. Specifically, block data from the left matrix tile (tile A) and the right matrix tile (tile B) are read according to a preset matrix calculation order and sent to the matrix calculation unit. The matrix calculation unit can process matrix multiplication operations of two blocks per clock cycle.

[0140] In this embodiment, it is assumed that the left matrix block tileA is a sparse matrix block. During the calculation process, the matrix calculation unit selects the corresponding block in tileB to participate in the operation according to the mask signal corresponding to tileA; since all-zero blocks in tileA have been pre-removed, the base address of the block in tileA can be directly linearly accumulated to achieve sequential reading of data.

[0141] Specifically, in one possible implementation, combining Figure 7 The diagram illustrates a matrix block computation process. This embodiment uses a one-sided sparse mode for the input data, which is mainly used in matrix multiplication (Matmul) operators.

[0142] The above step S501 specifically includes: obtaining the first dense dataset from the first matrix block according to the column order based on the first sparse indication information; correspondingly, obtaining the second dense data block set in the second matrix block includes: obtaining the second dense dataset from the second matrix block according to the row order.

[0143] In this embodiment, taking the first matrix block as a sparse matrix block as an example, one part is sparse data blocks and the other part is dense data blocks. Therefore, in the form of sparse memory access, memory access is performed at the data block granularity according to the first sparse indication information, and the first dense dataset is obtained from the first matrix block in column order. The first dense dataset only includes dense data blocks and does not include sparse data blocks.

[0144] Taking the second matrix block as a dense matrix block as an example, the entire matrix block needs to be read. Therefore, the second dense dataset is obtained from the second matrix block in row order. This dense dataset includes at least one dense data block.

[0145] Combination Figure 8 The diagram illustrates a matrix block memory access scheme, where M... K and K N represents the size of the first matrix block. During calculation, the first matrix block is calculated at the data block granularity along the column direction, and the second matrix block is calculated along the row direction, using the outer product method.

[0146] In one possible implementation, obtaining the first dense data block set in the first matrix block includes: determining the base address of multiple dense data blocks in the first dense dataset according to the first sparse indication information; and obtaining the first dense dataset in the first matrix block based on the base address of the multiple dense data blocks in the first dense dataset.

[0147] It should be noted that the first sparse indicator information only needs to store the base address and optional block length of each dense data block, without storing the coordinates of each non-zero element. For example, when the block size is 16 columns, a 1024-column matrix block only requires 6 bits to represent a block number, and the storage overhead of the indicator information is 1-2 orders of magnitude lower than that of the element-level sparse format. At the same time, base address addressing is the easiest operation to implement in hardware, without the need for complex index calculations, hash lookups, or out-of-order execution logic, which will not increase the complexity of the PE unit or affect the chip's operating frequency.

[0148] See Figure 9 The diagram illustrates another memory access method. In another embodiment of this application, dense memory access is used to implement data transfer for matrix multiplication operations. The specific process is as follows: When the matrix stored in HBM is still the original complete size, the data transfer in the first stage (from HBM to the chip's internal cache) does not require any modification. The corresponding tile blocks are directly transferred to the chip's internal cache according to the original matrix block size.

[0149] During the second stage of data transfer (from the internal cache to the matrix calculation unit), the read address of the block needs to be dynamically updated according to the mask signal. In this embodiment, it is also assumed that the left matrix block tileA is a sparse matrix block. That is, according to the mask signal corresponding to tileA, the non-zero block in tileA and the corresponding block in tileB are read and sent to the matrix calculation unit for operation.

[0150] In this embodiment, the method of reading the block of tile B is consistent with the previous embodiment. However, unlike the previous embodiment, the block address of tile A cannot be directly accumulated linearly. Instead, the address of non-zero block needs to be dynamically calculated based on the mask signal corresponding to tile A, and all-zero block is automatically skipped.

[0151] Specifically, in one possible implementation, such as Figure 10 The diagram shows another matrix block computation process. Step S501 specifically includes: obtaining a first dense data block set in the first matrix block, including: obtaining all data blocks in the first matrix block; and obtaining the first dense dataset from all data blocks according to column order based on the first sparse indication information.

[0152] Specifically, in the form of dense memory access, all data blocks in the first matrix block and the second matrix block of the second matrix block are obtained respectively, wherein all data blocks include the first dense dataset.

[0153] Since the first matrix block is in a sparse mode, with one part being sparse data blocks and the other part being dense data blocks, the first dense dataset is obtained from the first matrix block in column order based on the first sparsity indication information. Correspondingly, obtaining the second dense data block set in the second matrix block includes obtaining the second dense dataset from the second matrix block in row order.

[0154] During calculation, the first matrix block is divided into columns and the second matrix block is divided into rows. The matrix calculation is performed using the outer product method.

[0155] In one possible implementation, see Figure 11 The diagram illustrates a matrix multiplication operation. To further reduce memory access overhead during matrix multiplication, a double buffer is used to store i dense data blocks in the first matrix tile A. Since each dense data block in each column of the first matrix tile A needs to be externally multiplied by all data blocks in the corresponding row direction of the second matrix tile B, storing i dense data blocks in the double buffer allows for simultaneous calculation with the stored i dense data blocks by reading the corresponding data block of the second matrix tile B only once. This reduces the number of data accesses to the second matrix tile B to 1 / i of the original, significantly reducing memory overhead caused by repeated data movement.

[0156] Specifically, when i is greater than or equal to 2, that is, when there are more than two dense data blocks in the Nth column of the first dense data block set in the first matrix block tileA, each dense data block corresponds to all data blocks in the corresponding row direction of the second matrix block tileB. Therefore, i dense data blocks are stored in the register double buffer. Each time the data block of the second matrix block tileB is read in, it is calculated with the i dense data blocks stored in tileA. This is equivalent to reducing the data access of tileB by half, thus reducing the access overhead.

[0157] Meanwhile, when processing the last block of tile B, the next dense data block of the first matrix segment tile A can be preloaded into the register double buffer, thereby hiding the data loading delay of the next round of calculation and further reducing the overall memory access overhead.

[0158] In one possible implementation, see Figure 12The diagram illustrates another matrix multiplication operation. When i equals 1, it indicates that there is only one dense data block in the Nth column of the first matrix block tileA. In this case, this dense data block needs to be multiplied by all dense data blocks in the corresponding row direction of the second matrix block tileB. Therefore, the calculation can be completed simply by continuously accessing and reading all dense data blocks in the corresponding row direction of the second matrix block tileB.

[0159] In one possible implementation, when i equals 0, there are no dense data blocks in the Nth column of tileA. In this case, the matrix multiplication operation between the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set is skipped, further reducing the overall memory access overhead.

[0160] In one possible implementation, in order to further eliminate the memory access overhead during column switching and ensure that the computing resources of the computing unit continue to operate at full load, before obtaining the calculation result of the second matrix corresponding to the Nth column, j dense data blocks in the N+1th column of the first dense data block set are obtained in advance, and each dense data block in the Mth row of the second dense data block set is obtained, where j is a positive integer; After obtaining the calculation result of the second matrix corresponding to the Nth column, without waiting for data loading, matrix multiplication is performed on the j dense data blocks in the N+1th column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the calculation result of the third matrix corresponding to the N+1th column of the first dense data block set.

[0161] Specifically, see Figure 13 The diagram illustrates another matrix multiplication operation, using the Nth and (N+1)th columns of the first matrix block tile A and the Mth row of the second matrix block tile B as examples. The Nth column of the first matrix block tile A comprises two dense data blocks, Ablcok. i (Ab) i ) and Ablock i+1 (Ab) i+1 The N+1th column contains a dense data block Ablock. i+2 (Ab) i+2 The second matrix block consists of four dense data blocks, namely Bblcok j (Bb) j ), Bblcok j+1 (Bb) j+1 ), Bblcok j+2 (Bb) j+2 ) and Bblcok j+3 (Bb) j+3 ).

[0162] During the calculation process, the following calculation Ab is performed. i Bb j Ab i+1 Bb j Ab i+1 Bb j+1 Ab i Bb j+1 Ab i Bb j+2 Ab i+1 Bb j+2 Ab i+1 Bb j+3 Ab i Bb j+3 Before obtaining the calculation result of the second matrix corresponding to the Nth column, the j dense data blocks Ablock in the (N+1)th column of the first matrix block are obtained in advance. i+2 (Ab) i+2 The system obtains the dense data block of the Mth row of the second matrix block, thereby eliminating the memory access overhead during column switching and ensuring the continuous operation of the computing unit's computing resources.

[0163] After obtaining the calculation result of the second matrix corresponding to the Nth column, without waiting for data loading, calculate Ab respectively. i+2 Bb j Ab i+2 Bb j+1 Ab i+2 Bb j+2 Ab i+2 Bb j+3 This is used to obtain the calculation result of the third matrix corresponding to the (N+1)th column.

[0164] It should be understood that when j is greater than or equal to 2, the j dense data blocks can be stored in a register double buffer. Each time a data block of the second matrix segment tileB is read, it can be calculated with the stored j dense data blocks at the same time, thereby reducing the number of data accesses of the second matrix segment tileB to 1 / j of the original, and further reducing memory access overhead.

[0165] In another possible implementation, see Figure 14The diagram illustrates another matrix multiplication operation, using the Nth and N+1th columns of the first matrix block tile A and the Mth and M+1th rows of the second matrix block tile B as examples. The Nth column of the first matrix block tile A includes a dense data block Ablcok. i (Ab) i The N+1th column includes a dense data block and an Ablock. i+1 (Ab) i+1 The second matrix block consists of row M and row M+1. Row M contains four dense data blocks, namely Bblcok. j (Bb) j ), Bblcok j+1 (Bb) j+1 ), Bblcok j+2 (Bb) j+2 ) and Bblcok j+3 (Bb) j+3 The M+1th row contains four dense data blocks, namely Bblcok k (Bb) k ), Bblcok k+1 (Bb) k+1 ), Bblcok k+2 (Bb) k+2 ) and Bblcok k+3 (Bb) k+3 ).

[0166] Calculate Ab respectively i Bb j Ab i Bb j+1 Ab i Bb j+2 Ab i Bb j+3 Before obtaining the calculation result of the second matrix corresponding to the Nth column, one dense data block in the N+1th column of the first matrix block is read in advance, and the dense data block in the M+1th row of the second matrix block is obtained, thereby eliminating the memory access overhead during column switching and ensuring the continuous operation of the computing unit's computing resources.

[0167] After obtaining the calculation result of the second matrix corresponding to the Nth column, without waiting for data loading, calculate Ab respectively. i+1 Bb k Ab i+1 Bb k+1 Abi+1 Bbk +2 Ab i+1 Bb k+3 This is used to obtain the calculation result of the third matrix corresponding to the (N+1)th column.

[0168] Similarly, when j is greater than or equal to 2, the j dense data blocks can be stored in a register double buffer. Each time a data block of the second matrix segment tileB is read, it can be calculated simultaneously with the stored j dense data blocks, thereby reducing the number of data accesses of the second matrix segment tileB to 1 / j of the original, and further reducing memory access overhead.

[0169] Therefore, by pipelined overlap of the data loading process of column N+1 with the calculation process of column N, the implementation of this application can eliminate the memory access overhead during column switching, ensuring the continuous operation of the computing unit. Simultaneously, by combining register double buffering technology, the number of repeated data transfers in the second matrix block is further reduced, improving the overall efficiency of matrix multiplication operations and memory bandwidth utilization.

[0170] In one possible implementation, to adapt to the computational characteristics of low-precision FlashAttention scenarios, see [link to relevant documentation]. Figure 15 The diagram illustrates another matrix block computation process, and the data processing method further includes the following steps: The fifth dense data set is obtained from the third matrix block in row order, and the fifth dense data set includes multiple dense data blocks; The third matrix block has a different segmentation dimension than the first matrix block; Based on the second sparse indication information, the sixth dense data block set is obtained from the fourth matrix block in column order. The sixth dense data set includes at least one dense data block. The fourth matrix block has a different partitioning dimension than the second matrix block. The values ​​of multiple elements in the second sparse indication information are used to indicate the sparsity state of the third matrix calculation result obtained by performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set. Each dense data block in the fifth dense data block set is multiplied by each dense data block in the sixth dense data block set to obtain the calculation result of the third matrix.

[0171] Specifically, such as Figure 16The diagram shows a matrix block partitioning scheme. In a low-precision FlashAttention scenario, since both the third matrix block Q and the fourth matrix block K are dense, the fifth dense data block set is obtained from the third matrix block in row order, and the sixth dense data block set is obtained from the fourth matrix block in column order.

[0172] Specifically, the third matrix block has a different partitioning dimension than the first matrix block, and the fourth matrix block has a different partitioning dimension than the second matrix block. The first and second matrix blocks divide data blocks using two axes, while the second and third matrix blocks divide data blocks using only one axis.

[0173] The values ​​of multiple elements in the second sparse indication information are used to indicate the sparse state of the third matrix calculation result obtained by performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set.

[0174] It should be understood that, using the general matrix multiplication operation A Taking B=C as an example, where A is the left matrix, B is the right matrix, and C is the output matrix. In this embodiment, the aforementioned first sparsity indicator information is used to indicate the sparsity state of the first matrix block in the left matrix; the second sparsity indicator information can also be understood as being used to indicate the sparsity state of the fourth matrix block in the right matrix.

[0175] Therefore, by adjusting the segmentation dimension and data acquisition order of the matrix blocks according to the data flow characteristics of low-precision FlashAttention scenarios, this embodiment can be deeply integrated with the block calculation logic of FlashAttention, thereby improving the overall efficiency of matrix multiplication operations in this scenario.

[0176] As shown in Table 1, in one possible implementation, this application also provides a hardware-level sparse matrix multiplication instruction interface for AI acceleration chip computing cores (assuming block granularity is B and tile granularity is M). K and K N), specifically as follows:

[0177] Table 1

[0178] As can be seen from Table 1, the embodiments of this application do not require complex element-level index parsing, hash lookup or out-of-order execution logic, but only simple base address calculation and sparse mask matrix judgment to complete sparse acceleration.

[0179] Moreover, the same instruction interface can cover the matrix multiplication calculation needs of AI scenarios such as computer vision, recommendation systems, and multimodal models, without the need to design dedicated instructions for different scenarios.

[0180] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0181] To facilitate better implementation of the above-described solutions in the embodiments of this application, related apparatus for implementing the above-described solutions is also provided below.

[0182] This application provides a data processing apparatus. The data processing apparatus can be applied to a processor.

[0183] The data processing device includes: an acquisition unit and a matrix calculation unit; wherein: The acquisition unit is used to acquire a first dense data block set in a first matrix block; wherein, the first dense data set includes multiple dense data blocks, the first matrix block belongs to a first matrix, and the first matrix includes multiple matrix blocks including the first matrix block; The acquisition unit is further configured to acquire a second dense data block set in the second matrix block; wherein the second dense data set includes at least one dense data block, the second matrix block belongs to a second matrix, and the second matrix includes multiple matrix blocks including the second matrix block; The matrix calculation unit is used to perform matrix multiplication operations on each dense data block in the first dense data block set and each dense data block in the second dense data block set to obtain the first matrix calculation result.

[0184] In one possible implementation, the first matrix block also includes sparse data blocks.

[0185] In one possible implementation, the acquisition unit is further configured to acquire a third dense data block set for each of the remaining matrix blocks in the first matrix; Obtain the set of the fourth dense data blocks for each of the remaining matrix blocks in the second matrix; The matrix calculation unit is further configured to perform matrix multiplication operations on each dense data block in the third dense data block set and each dense data block in the fourth dense data block set, so as to obtain the matrix calculation result corresponding to each of the remaining matrix blocks.

[0186] In one possible implementation, the acquisition unit is specifically used to acquire the first dense dataset from the first matrix in column order based on the first sparse indication information. Accordingly, obtaining the second dense data block set in the second matrix partition includes: The second dense dataset is obtained from the second matrix blocks in row order.

[0187] In one possible implementation, the acquisition unit is specifically used to determine the base address of multiple dense data blocks in the first dense dataset based on the first sparse indication information. Based on the base addresses of multiple dense data blocks in the first dense dataset, the first dense dataset in the first matrix block is obtained.

[0188] In one possible implementation, the first sparsity indication information is a sparse masking matrix, and the values ​​of multiple elements in the sparse masking matrix are used to indicate the sparse state of the first matrix block.

[0189] In one possible implementation, the matrix calculation unit is specifically used to perform matrix multiplication operations based on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set, so as to obtain the second matrix calculation result corresponding to the Nth column. Wherein, N, M, and i are positive integers.

[0190] In one possible implementation, the data processing device further includes: a storage unit; The i equals 1; or, The i is greater than or equal to 2, and the storage unit is used to store the i dense data blocks in the Nth column of the first dense data block set in the register.

[0191] In one possible implementation, if the Nth column of the first matrix block does not include a dense data block, the following step is skipped: performing matrix multiplication based on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set.

[0192] In one possible implementation, i is greater than or equal to 2; Before obtaining the calculation result of the second matrix corresponding to the Nth column, the acquisition unit is specifically used to acquire j dense data blocks in the N+1th column of the first dense data block set, and to acquire each dense data block in the Mth row of the second dense data block set, where j is a positive integer; After obtaining the second matrix calculation result corresponding to the Nth column, the matrix calculation unit is further used to perform matrix multiplication operations on the j dense data blocks in the N+1th column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the third matrix calculation result corresponding to the N+1th column of the first dense data block set.

[0193] In one possible implementation, j equals 1; or, The storage unit, where j is greater than or equal to 2, is used to store the j dense data blocks in the (N+1)th column of the first dense data block set in a register.

[0194] In one possible implementation, the acquisition unit is further configured to acquire a fifth dense data block set from the third matrix block in row order, the fifth dense data set comprising multiple dense data blocks; the third matrix block has a different partitioning dimension than the first matrix block. Based on the second sparse indication information, the sixth dense data block set is obtained from the fourth matrix block in column order. The sixth dense data set includes at least one dense data block. The fourth matrix block has a different partitioning dimension than the second matrix block. The values ​​of multiple elements in the second sparse indication information are used to indicate the sparsity state of the third matrix calculation result obtained by performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set. The matrix calculation unit is also used to perform matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set to obtain the third matrix calculation result.

[0195] This application provides a computing device 160. For example... Figure 17 As shown, the computing device 160 includes a bus 162, a processor 164, a memory 166, and a communication interface 168. The processor 164, the memory 166, and the communication interface 168 communicate with each other via the bus 162. The computing device 160 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 160.

[0196] Bus 162 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 17The bus 162 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 162 may include a path for transmitting information between various components of the computing device 160 (e.g., memory 166, processor 164, communication interface 168).

[0197] Processor 164 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0198] Memory 166 may include volatile memory, such as random access memory (RAM). Processor 164 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0199] The memory 166 stores executable program code, and the processor 164 executes the executable program code to implement the functions of the aforementioned acquisition unit and matrix calculation unit, thereby realizing the data processing method applied to the computing device cluster in the above embodiment. That is, the memory 166 stores instructions for executing the data processing method applied to the computing device cluster in the above embodiment.

[0200] The communication interface 168 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 160 and other devices or communication networks.

[0201] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0202] like Figure 18 As shown, the computing device cluster includes at least one computing device 160. The memory 166 of one or more computing devices 160 in the computing device cluster may store the same instructions for performing data processing methods.

[0203] In some possible implementations, the memory 166 of one or more computing devices 160 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 160 can jointly execute instructions for executing data processing methods.

[0204] It should be noted that the memory 166 in different computing devices 160 within the computing device cluster can store different instructions, each used to execute a portion of the data processing method's functions. That is, the instructions stored in the memory 166 of different computing devices 160 can implement one or more functions of the acquisition unit.

[0205] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN), a local area network (LAN), or similar. Figure 19 One possible implementation is shown. For example... Figure 19 As shown, two computing devices 160A and 160B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 166 in computing device 160A may store instructions for executing the functions of the first processing module. Simultaneously, the memory 166 in computing device 160B may store instructions for executing the functions of the second processing module. Alternatively, the memory 166 in computing device 160A may store instructions for executing some functions of the second processing module, while the memory 166 in computing device 160B may store instructions for executing another part of the functions of the second processing module, and so on.

[0206] It should be understood that Figure 19 The functions of the computing device 160A shown can also be performed by multiple computing devices 160. Similarly, the functions of the computing device 160B can also be performed by multiple computing devices 160.

[0207] In some possible implementations, the memory 166 of one or more computing devices 160 in the computing device cluster may also store partial instructions for executing data processing methods. In other words, a combination of one or more computing devices 160 can jointly execute instructions for executing data processing methods.

[0208] It should be noted that the memory 166 in different computing devices 160 within the computing device cluster can store different instructions for executing parts of the data processing methods. That is, the instructions stored in the memory 166 of different computing devices 160 can implement one or more functions of the processing module.

[0209] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method.

[0210] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to perform a data processing method.

[0211] This application also provides a chip system including a processor for implementing the steps performed by the aforementioned computing device cluster. In one possible design, the chip system may further include a memory for storing necessary program instructions and data. This chip system may be composed of chips or may include chips and other discrete devices.

[0212] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0213] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0214] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0215] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0216] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A data processing method, characterized in that, The method includes: Obtain the first dense data block set in the first matrix block; wherein, the first dense data set includes multiple dense data blocks, the first matrix block belongs to the first matrix, and the first matrix includes multiple matrix blocks including the first matrix block; Obtain the second dense data block set in the second matrix block; wherein the second dense data set includes at least one dense data block, the second matrix block belongs to the second matrix, and the second matrix includes multiple matrix blocks including the second matrix block; Each dense data block in the first dense data block set is multiplied by each dense data block in the second dense data block set to obtain the first matrix calculation result.

2. The method according to claim 1, characterized in that, The first matrix block also includes sparse data blocks.

3. The method according to claim 1, characterized in that, The method further includes: Obtain the third dense data block set for each of the remaining matrix blocks in the first matrix; Obtain the set of the fourth dense data blocks for each of the remaining matrix blocks in the second matrix; Each dense data block in the third dense data block set is multiplied by each dense data block in the fourth dense data block set to obtain the matrix calculation result corresponding to each of the remaining matrix blocks.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the first dense data block set in the first matrix block includes: Based on the first sparse indication information, the first dense dataset is obtained from the first matrix in column order by dividing it into blocks; Accordingly, obtaining the second dense data block set in the second matrix partition includes: The second dense dataset is obtained from the second matrix blocks in row order.

5. The method according to claim 4, characterized in that, The step of obtaining the first dense data block set in the first matrix block includes: The base addresses of multiple dense data blocks in the first dense dataset are determined based on the first sparse indication information; Based on the base addresses of multiple dense data blocks in the first dense dataset, the first dense dataset in the first matrix block is obtained.

6. The method according to claim 4, characterized in that, The step of obtaining the first dense data block set in the first matrix block includes: Obtain all data blocks in the first matrix block; The first dense dataset is obtained from all data blocks in column order based on the first sparse indication information.

7. The method according to claim 4, characterized in that, The first sparsity indication information is a sparse masking matrix, and the values ​​of multiple elements in the sparse masking matrix are used to indicate the sparse state of the first matrix block.

8. The method according to any one of claims 1 to 3, characterized in that, Each dense data block in the first dense data block set is multiplied by each dense data block in the second dense data block set to obtain the first matrix calculation result, including: Matrix multiplication is performed on the i dense data blocks in the Nth column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the calculation result of the second matrix corresponding to the Nth column. Wherein, N, M, and i are positive integers.

9. The method according to claim 8, characterized in that, The i equals 1; or, If i is greater than or equal to 2, the i dense data blocks in the Nth column of the first dense data block set are stored in the register.

10. The method according to claim 8, characterized in that, Where i is greater than or equal to 2, the method further includes: Before obtaining the calculation result of the second matrix corresponding to the Nth column, obtain j dense data blocks in the N+1th column of the first dense data block set, and obtain each dense data block in the Mth row of the second dense data block set, where j is a positive integer; After obtaining the second matrix calculation result corresponding to the Nth column, matrix multiplication is performed on the j dense data blocks in the N+1th column of the first dense data block set and each dense data block in the Mth row of the second dense data block set to obtain the third matrix calculation result corresponding to the N+1th column of the first dense data block set.

11. The method according to claim 10, characterized in that, The j equals 1; or, If j is greater than or equal to 2, the j dense data blocks in the (N+1)th column of the first dense data block set are stored in a register.

12. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The fifth dense data set is obtained from the third matrix block in row order, and the fifth dense data set includes multiple dense data blocks; the third matrix block has a different partitioning dimension than the first matrix block. Based on the second sparse indication information, the sixth dense data block set is obtained from the fourth matrix block in column order. The sixth dense data set includes at least one dense data block. The fourth matrix block has a different partitioning dimension than the second matrix block. The values ​​of multiple elements in the second sparse indication information are used to indicate the sparsity state of the third matrix calculation result obtained by performing matrix multiplication operations between each dense data block in the fifth dense data block set and each dense data block in the sixth dense data block set. Each dense data block in the fifth dense data block set is multiplied by each dense data block in the sixth dense data block set to obtain the calculation result of the third matrix.

13. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire a first dense data block set in a first matrix block; wherein, the first dense data set includes multiple dense data blocks, the first matrix block belongs to a first matrix, and the first matrix includes multiple matrix blocks including the first matrix block; The acquisition unit is further configured to acquire a second dense data block set in the second matrix block; wherein the second dense data set includes at least one dense data block, the second matrix block belongs to a second matrix, and the second matrix includes multiple matrix blocks including the second matrix block; The matrix calculation unit is used to perform matrix multiplication operations on each dense data block in the first dense data block set and each dense data block in the second dense data block set to obtain the first matrix calculation result.

14. The apparatus according to claim 13, characterized in that, The first matrix block also includes sparse data blocks.

15. The apparatus according to claim 13, characterized in that, The acquisition unit is further configured to acquire the third dense data block set of each matrix block of the remaining matrix blocks in the first matrix; Obtain the set of the fourth dense data blocks for each of the remaining matrix blocks in the second matrix; The matrix calculation unit is further configured to perform matrix multiplication operations on each dense data block in the third dense data block set and each dense data block in the fourth dense data block set, so as to obtain the matrix calculation result corresponding to each of the remaining matrix blocks.

16. The apparatus according to any one of claims 13 to 15, characterized in that, The acquisition unit is specifically used to acquire the first dense dataset from the first matrix in block order based on the first sparse indication information; Accordingly, obtaining the second dense data block set in the second matrix partition includes: The second dense dataset is obtained from the second matrix blocks in row order.

17. The apparatus according to claim 16, characterized in that, The acquisition unit is specifically used to determine the base address of multiple dense data blocks in the first dense dataset based on the first sparse indication information. Based on the base addresses of multiple dense data blocks in the first dense dataset, the first dense dataset in the first matrix block is obtained.

18. The apparatus according to claim 16, characterized in that, The first sparsity indication information is a sparse masking matrix, and the values ​​of multiple elements in the sparse masking matrix are used to indicate the sparse state of the first matrix block.

19. A processor, characterized in that, The processor is configured to execute computer programs or instructions in memory to perform the data processing method as described in any one of claims 1 to 12.

20. A computer-readable storage medium, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the data processing method as described in any one of claims 1 to 12.

21. A computer program product, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the data processing method as described in any one of claims 1 to 12.

Images