An operator acceleration method and related devices

Abstract

An operator acceleration method is applied to a computing card, which includes multiple artificial intelligence (AI) cores and global memory. The method includes: acquiring a computational task to be executed, the task information including input data for a scattering operator, at least one source data, and index data corresponding to at least one source data; determining the target core corresponding to each index from the multiple AI cores; and ensuring that indices indicating the same position in the index data correspond to the same target core. Then, the source data corresponding to each index is moved into the target core. A first computational task is executed within the target core based on the moved source data and input data. Subsequently, a second computational task is executed between the target cores based on intermediate results to obtain the operator execution result. This method introduces a core-sharing scheme where the same index corresponds to the same core, ensuring that source data corresponding to indices indicating the same position are preferentially moved to the same AI core for computation, preventing the computing card from degenerating into using a single core, improving multi-core parallelism, and enhancing end-to-end performance.

Description

Technical Field

[0001] This application relates to the field of computing technology, and in particular to an operator acceleration method, an operator acceleration device, a computing card, a computing cluster, a computer-readable storage medium, and a computer program product. Background Technology

[0002] As the complexity and scale of computing tasks continue to increase, traditional single-card (or simply single-card) architectures are insufficient to meet computing demands. Therefore, multiple computing cards can be used to complete computing tasks. For example, in artificial intelligence (AI) scenarios, multiple computing cards can form a training cluster to train large-scale AI models, or they can form an inference cluster to perform inference tasks using trained AI models. As another example, in high-performance computing (HPC) scenarios, multiple computing cards are connected through various interconnect technologies to form an HPC cluster. The HPC cluster processes massive multidimensional datasets in parallel and solves complex problems at extremely high speeds. These complex problems can include, but are not limited to, climate modeling, protein folding, and fluid behavior modeling.

[0003] Multiple computing cards typically require collective communication to complete computational tasks. For example, in AI scenarios, collective communication is often needed between computing cards to achieve gradient synchronization. During the execution of computational tasks, computing cards widely use scatter operators to facilitate collective communication between cards.

[0004] However, the end-to-end time consumption of scattering operators accounts for a large proportion, gradually becoming a performance bottleneck for the cluster. Summary of the Invention

[0005] This application provides an operator acceleration method that introduces a core-sharing scheme where the same index corresponds to the same core. This ensures that source data corresponding to indices pointing to the same location are preferentially transferred to the same AI core for computation, preventing multi-core computing cards from degenerating into single-core computation when executing scattering operators. This method enables multi-core concurrent processing while maintaining accuracy, improving multi-core parallelism, shortening end-to-end latency, and enhancing end-to-end performance. This application also provides an operator acceleration device, computing card, computing cluster, computer-readable storage medium, and computer program product corresponding to the above method.

[0006] Firstly, this application provides an operator acceleration method. This method is applied to a computing card. The computing card includes multiple AI cores and global memory. When the matrix computation unit and vector computation unit in the AI ​​core are coupled, the multiple AI cores can include multiple complete AI cores, each AI core including a vector computation unit and a matrix computation unit. When the matrix computation unit and vector computation unit in the AI ​​core are separated, the multiple AI cores can include multiple vector computation cores.

[0007] Specifically, the computing card can acquire computational tasks to be executed. The task information includes input data for a scattering operator, at least one source data set, and index data corresponding to at least one source data set. The input data is the data to be processed using the source data, and the index data indicates the position of the data to be processed within the input data. The computing card can determine the target kernel corresponding to each index in the index data from multiple AI kernels. Indices indicating the same position in the index data correspond to the same target kernel. The computing card can move the source data corresponding to each index in the index data into the target kernel corresponding to each index. Based on the source data and input data moved into the target kernel, it executes a first computational task within the target kernel to obtain intermediate results. These intermediate results can be moved into global memory. Then, based on the intermediate results, a second computational task is executed between the target kernels to obtain the operator execution result.

[0008] This method introduces a core-sharing scheme where indexes at the same location are processed on the same AI core. This ensures that source data corresponding to indexes at the same location are preferentially moved to the same AI core for computation. Thus, even if the input data is floating-point or the computation task is deterministic, it avoids reverting to single-core computation. It enables multi-core concurrent processing while maintaining accuracy, improving multi-core parallelism, shortening end-to-end latency, and enhancing end-to-end performance. Furthermore, by merging index data, the intermediate results obtained from moving indexes at the same location and their corresponding source data to the same AI core only need to be moved to global memory once. For scenarios with a large number of duplicate indexes, this method can significantly reduce MTE3 data transfer time and effectively improve the bandwidth utilization of the computing card.

[0009] In some possible implementations, the compute card can identify the same target core from multiple AI cores for indices indicating the same position in the index data. On the one hand, this allows the source data corresponding to indices indicating the same position to be preferentially moved to the same AI core for computation, avoiding degradation to single-core computation and improving multi-core parallelism; on the other hand, it can also reduce the number of times the target core is determined, reducing the overhead of core partitioning.

[0010] In some possible implementations, when the computation task is deterministic, the computing card can identify the same target core from multiple AI cores based on indices indicating the same location in the index data. "Deterministic computation" refers to a computation method where, under the same initial conditions, the computation result is always the same regardless of how many times the computation process is repeated. In other words, given the same input data and computing environment (such as the same hardware), deterministic computation will produce completely consistent outputs.

[0011] This method, by employing a core-sharing strategy with the same core and index for computational tasks that enable deterministic computation, can ensure the consistency of computation results during multiple repeated computations, thus meeting business requirements.

[0012] In some possible implementations, when the precision of the floating-point type of the input data does not meet the task objective of the computation, the computing card can convert the floating-point type of the input data from a first floating-point type to a second floating-point type to obtain the converted input data. The precision of the second floating-point type is higher than that of the first floating-point type. Accordingly, the computing card can execute the first computation task within the target core based on the source data loaded into the target core and the converted input data.

[0013] This method improves accuracy by converting low-precision floating-point data that does not meet the task objectives, thereby ensuring the accuracy of the calculation results and meeting business requirements.

[0014] In some possible implementations, the computing card can deduplicate the index data, obtain the occurrence count of each index in the deduplicated index data, and then determine the target core corresponding to each index in the deduplicated index data from multiple AI cores based on the occurrence count of each index in the deduplicated index data.

[0015] This method divides the index into cores according to the number of times each index appears in the deduplicated index data, which can determine a reasonable target core for each index and improve the overall concurrency as much as possible.

[0016] In some possible implementations, the computing card can sort the deduplicated index data based on the frequency of each index occurrence, resulting in an ordered index set. Then, the computing card can determine the target core corresponding to each index in the ordered index set based on the amount of task to be processed by at least one AI core among multiple AI cores.

[0017] This allows the deduplicated index data to be distributed across AI cores based on the amount of tasks to be processed by each AI core, ensuring that the workload processed by each AI core is as balanced as possible.

[0018] In some possible implementations, the computing card can determine the target core corresponding to each index in the ordered index set through load balancing based on the amount of tasks to be processed by at least one AI core among multiple AI cores.

[0019] This method improves the concurrency of multiple AI cores and enhances operator execution performance by employing a load balancing strategy to distribute the indexes across AI cores.

[0020] In some possible implementations, the computing card is a Single Instruction Multiple Data (SIMD) chip. SIMD chips can process multiple data with a single instruction, and naturally have a good affinity for scattering operators or scattering operations, which is beneficial for accelerating scattering operators.

[0021] Secondly, this application provides an operator acceleration device. The device is deployed on a computing card, which includes multiple artificial intelligence (AI) cores and global memory. The device includes:

[0022] The task acquisition module is used to acquire the computation task to be executed. The task information of the computation task includes input data of the scattering operator, at least one source data and index data corresponding to the at least one source data. The input data is the data to be processed using the source data, and the index data is used to indicate the position of the data to be processed in the input data.

[0023] The core-splitting module is used to determine the target core corresponding to each index in the index data from the plurality of AI cores, wherein the target cores corresponding to indexes indicating the same position in the index data are the same;

[0024] The task execution module is used to move the source data corresponding to each index in the index data into the target core corresponding to each index, and to perform a first calculation task in the target core according to the source data moved into the target core and the input data to obtain intermediate results;

[0025] The task execution module is also used to move the intermediate results into the global memory, and execute a second computation task between the target cores based on the intermediate results to obtain the operator execution result.

[0026] In some possible implementations, the core-splitting module is specifically used for:

[0027] For indexes indicating the same position in the index data, the same target core is determined from the plurality of AI cores.

[0028] In some possible implementations, the core-splitting module is specifically used for:

[0029] When the computation type of the computation task is deterministic computation, for the indexes in the index data that indicate the same position, the same target core is determined from multiple AI cores.

[0030] In some possible implementations, the task execution module is specifically used for:

[0031] When the precision of the floating-point type of the input data does not meet the task objective of the computation task, the floating-point type of the input data is converted from a first floating-point type to a second floating-point type to obtain the converted input data. The precision of the second floating-point type is higher than that of the first floating-point type.

[0032] A first computational task is performed within the target core based on the source data loaded into the target core and the transformed input data.

[0033] In some possible implementations, the core-splitting module is specifically used for:

[0034] The index data is deduplicated, and the number of occurrences of each index in the deduplicated index data is obtained;

[0035] Based on the number of times each index appears in the deduplicated index data, the target core corresponding to each index in the deduplicated index data is determined from the plurality of AI cores.

[0036] In some possible implementations, the core-splitting module is specifically used for:

[0037] Based on the frequency of occurrence of each index in the deduplicated index data, the deduplicated index data is sorted to obtain an ordered index set;

[0038] Based on the amount of tasks to be processed by at least one of the multiple AI cores, the target core corresponding to each index in the ordered index set is determined.

[0039] In some possible implementations, the core-splitting module is specifically used for:

[0040] Based on the amount of tasks to be processed by at least one AI core among the multiple AI cores, the target core corresponding to each index in the ordered index set is determined through load balancing.

[0041] In some possible implementations, the computing card is a single instruction multiple data (SIMD) chip.

[0042] Thirdly, this application provides a computing card. The computing card includes multiple AI cores and global memory. For example, the computing card can be a neural processing unit (NPU). The AI ​​core includes matrix computation units, vector computation units, and scalar computation units designed for neural networks. When the AI ​​core adopts a separate architecture, the aforementioned vector computation units and matrix computation units can form a vector computation core and a matrix computation core. Multiple AI cores can include multiple vector computation cores. It should be noted that the computing card can also be a processor other than an NPU; the NPU is merely an example. The AI ​​core is used to execute computer-readable instructions loaded into the global memory to perform the operator acceleration method as described in the first aspect of this application or any implementation thereof.

[0043] Fourthly, this application provides a computing cluster. The computing cluster includes multiple computing cards, which can be connected via a bus or network. For example, in a single-machine multi-card (or one-host multi-computing-card) architecture, the multiple computing cards can be connected via a bus. As another example, in a multi-machine multi-card (or multi-host multi-computing-card) architecture, computing cards within the same computing device can be connected via a bus, while computing cards in different computing devices can be connected via a network.

[0044] Fifthly, this application provides a computer-readable storage medium storing instructions that instruct a computing card or computing cluster to execute the operator acceleration method described in the first aspect or any implementation thereof.

[0045] In a sixth aspect, this application provides a computer program product containing instructions that, when run on a computing card or computing cluster, causes the computing card or computing cluster to execute the operator acceleration method described in the first aspect or any implementation thereof.

[0046] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0047] To more clearly illustrate the technical methods of this application, the accompanying drawings used will be briefly described below.

[0048] Figure 1 A schematic diagram of the architecture of a computing card provided in this application;

[0049] Figure 2 This application provides a schematic diagram of data interaction during the execution of a scattering operator.

[0050] Figure 3 A flowchart of an operator acceleration method provided in this application;

[0051] Figure 4 A flowchart illustrating a data type conversion method provided in this application;

[0052] Figure 5 A flowchart illustrating an operator acceleration based on an index equalization strategy provided for this application;

[0053] Figure 6 A flowchart illustrating an operator acceleration strategy based on the same index and same core provided in this application;

[0054] Figure 7 This application provides a schematic diagram of the structure of an operator acceleration device;

[0055] Figure 8 This application provides a schematic diagram of the structure of a computing cluster;

[0056] Figure 9 This is a schematic diagram of another computing cluster structure provided in this application. Detailed Implementation

[0057] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0058] First, some technical terms involved in the embodiments of this application will be introduced.

[0059] Artificial intelligence (AI) is the ability to learn knowledge from external data using mathematical algorithms and then leverage that knowledge to flexibly achieve specific goals and tasks. Learning from external data can be achieved through machine learning (ML) and deep learning (DL). Deep learning is an algorithm that attempts to perform high-level abstractions of data using multiple processing layers containing complex structures or multiple nonlinear transformations. Deep learning can be further divided into supervised learning, semi-supervised learning, and unsupervised learning.

[0060] AI models are models built using AI technologies, including but not limited to machine learning models and deep learning models. AI models can be trained using large amounts of sample data through supervised, semi-supervised, or unsupervised learning algorithms. A trained AI model can then be used to perform inference tasks, such as image classification and text generation.

[0061] High-performance computing (HPC) is a computer cluster system that aggregates the computing power of multiple computing cards to enhance powerful computing performance. HPC connects multiple computing cards through various interconnect technologies, leveraging the combined computing power of the connected cards to handle large-scale computational problems. These large-scale computational problems can be computational tasks in fields such as scientific research, weather forecasting, simulation experiments, and image processing. Specifically, the management node of the HPC cluster decomposes the computational tasks and schedules the subtasks to different computing cards for computation. Computing cards are processors that provide computing power, and include, but are not limited to, central processing units (CPUs), neural processing units (NPUs), graphics processing units (GPUs), tensor processing units (TPUs), and other xPUs.

[0062] Collective communications is a global communication operation involving all processes (or threads) within a process group (or thread group). The processes or threads can be programs running on a computing card. Collective communications can include one-to-many broadcast operations, many-to-one gather operations, one-to-many scatter operations, many-to-one reduce operations, many-to-many global reduce operations, and combined reduce-scatter operations.

[0063] Whether in AI model training and inference scenarios or in HPC clusters handling large-scale computational problems, scattering operations (or scatter indexing operations, distribution operations, or scatter-type operations) are often used between multiple computing cards. For ease of understanding, the following example illustrates this using an AI model training scenario.

[0064] To address the challenges of training difficulty and resource consumption in large-scale AI models, researchers have proposed the Mixture of Experts (MoE) architecture. MoE, also known as a multi-expert model, is based on the core idea of ​​dividing a large neural network into multiple independent sub-models, each specializing in processing a specific type of input data. In a MoE model, each sub-model is called an expert, and a gating network is responsible for selecting the most suitable expert based on the input data. During the gating network's selection of the most appropriate expert for processing specific types of input data, scattering operators used for performing scattering operations are frequently invoked.

[0065] Scattering operators can include the `scatter` operator or its derivatives. Derivatives of the `scatter` operator include, but are not limited to, the `scatteradd`, `indexadd`, and `indexput` operators. The `scatteradd` operator is used to scatter and add data to a specified location. For example, in AI model training or inference scenarios, the `scatteradd` operator can be applied to automatic differentiation and tensor computation in deep learning frameworks. Another example is in HPC-based data processing or scientific computing scenarios, where the `scatteradd` operator can add multiple values ​​to a matrix according to their corresponding indices in the index data, thus constructing a complete sparse matrix. The `indexadd` operator is used to perform addition operations based on index data. For example, in AI model training scenarios, each parameter has an index indicating its position in the parameter list or parameter tensor. After backpropagation calculates the gradient, the `indexadd` operator can be used to add the gradient values ​​to the parameter tensor according to the parameter indices to update the AI ​​model's parameters and achieve AI model training. The `indexput` operator is used to place target data (e.g., values) to a specified location according to the index data. For example, in image data processing scenarios, the indexput operator is used to update pixel values ​​in an image.

[0066] For ease of description, the scatteradd operator is used as an example. The calculation process of the scatteradd operator can be represented as follows:

[0067] X[index] += Value (1)

[0068] Where X represents the input data, index represents the index data, and Value represents the value, also known as the source data, which in formula (1) refers to the data accumulated to the input data according to the index. X, index, and Value can be represented and stored in the form of tensors. Value can also be a scalar. It should be noted that a tensor is a one-dimensional or multi-dimensional array, an array is an ordered sequence of elements, and a scalar is a number that only has magnitude and no direction. In some cases, a scalar can be regarded as a special tensor, such as a zero-order tensor.

[0069] The scatteradd operator supports adding values ​​from the value tensor to the destination tensor (denoted as dest_tensor) along a specified dimension, based on the index of the index tensor.

[0070] In this example, the AI ​​model is a classification model, which can be used to perform classification tasks. When it is necessary to correct the prediction results of the classification task, the scatteradd operator in the deep learning framework can be used to correct the prediction results. The prediction results are represented by a prediction tensor, where the index tensor specifies the location for correction, and the correction value tensor specifies the correction value.

[0071] In the example, the prediction tensor has a shape of (3,5), indicating that there are 3 samples, each containing 5 class predictions. The prediction tensor can be initialized as an all-zero tensor.

[0072] The index tensor in the example has a shape of (3,3) and can be represented in the following matrix form:

[0073]

[0074] Each element in the index tensor specifies the position where the correction value is added. For example, `index_tensor[0,0] = 1` means that for the first sample, the correction value is added at the second class position (the index starts from 0, so it's at the second class position). Another example is `index_tensor[1,2] = 4`, which means that for the second sample, the correction value is added at the fifth class position.

[0075] The modified value tensor in the example has a shape of (3,3) and can be represented in the following matrix form:

[0076]

[0077] Each element in the correction value tensor (value_tensor) indicates the correction value to be added.

[0078] The scatteradd operator adds the correction values ​​from the correction value tensor to the prediction tensor along the i-th dimension (e.g., the 0th or 1st dimension) according to the index of the index tensor. Here, the 0th dimension can be a row, and the 1st dimension can be a column.

[0079] Let's illustrate this by adding correction values ​​to the prediction tensor along the first dimension (column direction). For the first row of the prediction tensor (the first sample), when index_tensor[0,0] = 1 and value_tensor[0,0] = 0.1, 0.1 can be added to the prediction_tensor[0,1] position (the column with index 1). When index_tensor[0,1] = 2 and value_tensor[0,1] = 0.2, 0.2 can be added to the prediction_tensor[0,2] position (the column with index 2). And so on. The prediction tensor after adding correction values ​​can be represented in the following matrix form:

[0080]

[0081] Implementing reliable and efficient scatter operations on computing cards plays a crucial role in improving the performance of MOE models on computing cards. The computing card can be a Single Instruction Multiple Data (SIMD) chip, which can process multiple data points with a single instruction, naturally possessing good affinity for scatter operators or operations.

[0082] See Figure 1 The diagram shows a hardware architecture of a computing card 10, which includes an AI core 100 and global memory 200. The AI ​​core 100, also known as the AI ​​Core, is the core of the computing card 10 and typically employs a Domain Specific Architecture (DSA) to adapt to common applications and algorithms in a specific domain. Global memory 200 is used to store the input, intermediate, or output data of the AI ​​core during computation.

[0083] AI Core 100 is responsible for executing computationally intensive operators related to scalars, vectors, and tensors. AI Core 100 includes several basic computational units: matrix (Cube) computation units, vector (Vector) computation units, and scalar (Scalar) computation units. These units perform different types of data computations. It should be noted that these different types of computational units form multiple independent execution pipelines; through unified scheduling and mutual cooperation, computational efficiency can be optimized.

[0084] Hardware architectures are categorized into coupled and separated architectures based on whether the matrix computation unit and vector computation unit are deployed on the same core. This application uses a separated architecture as an example. In the separated architecture, AI core 100 is split into matrix computation core 102 and vector computation core 104. Matrix computation core 102 is also called AI Cube, abbreviated as AIC, and vector computation core 104 is also called AI Vector, abbreviated as AIV. Matrix computation core 102 and vector computation core 104 are independent of each other, each having its own scalar computation unit and capable of independently loading its own code, thus achieving decoupling between matrix computation and vector computation. Figure 1 As shown, data can be transferred between matrix computation core 102 and vector computation core 104 via global memory 200. Among these, Figure 1 The computing card 10 in the document is a multi-core computing card, and therefore, it can include multiple vector computing cores 104. It should be noted that in a discrete architecture, the multi-core can also be multiple matrix computing cores 102, and in a coupled architecture, the multi-core can be multiple AI cores. This application does not impose any limitations on this.

[0085] AI Core 100 also includes storage units (such as hardware storage and data transfer units) and control units. AI Core 100 includes internal and external storage. Global memory 200 can serve as external storage for AI Core 100, also known as off-core storage. Memory storage can be buffers, including but not limited to L0 buffers, L1 buffers, and Unified Buffers (UB). The L0 buffer can be further divided into L0A, L0B, and L0C. AI Core 100 can load data from external storage into internal storage to complete corresponding computational tasks. It should be noted that in the separate architecture, matrix computation core 102, in addition to the existing L0 and L1 buffers, adds a bias table buffer (BT buffer) and a fixed pipe buffer (FP buffer). The BT buffer stores the bias of the AI ​​model, and the FP buffer stores quantization parameters and activation parameters (such as ReLU parameters).

[0086] To facilitate data transmission and handling within the AI ​​Core 100, the AI ​​Core 100 also includes a Bus Interface Unit (BIU), Memory Transfer Engine 1 (MTE1), Memory Transfer Engine 2 (MTE2), and Memory Transfer Engine 3 (MTE3). The BIU serves as the interface between the AI ​​Core and the bus; the MTEs are data transfer units that handle data transfer between different buffers. Figure 1 (Not shown in the image) is the interface between AI core 100 and the bus. MTE is for data transfer, which completes the data transfer between different buffers.

[0087] In the separate architecture, the matrix computation core 102 can include 5 parallel execution units (transfer units and computation units) and 7 memory units. The 5 parallel execution units can be MTE1, MTE2, MTE3, and the matrix computation unit. The 7 memory units include off-core global memory 200 (off-core memory) and L1 buffer, L0A, L0B, L0C, BT buffer, and FP buffer. The vector computation core 104 can include 4 parallel execution units and 2 memory units. The 4 parallel execution units can be MTE2, MTE3, the vector computation unit, and the scalar computation unit. The 2 memory units include global memory 200 and a unified buffer.

[0088] The data flow for vector computation can be represented as follows: data is moved from global memory 200 to a unified buffer; the vector computation unit reads data from the unified buffer to perform vector computation; and the computation result is then moved back to global memory 200. Therefore, the data flow for vector computation can be represented as GM-UB-[Vector]-UB-GM. Similarly, the data flow for matrix computation can be represented as follows: data is moved from global memory 200 to the L1 buffer, then from the L1 buffer to L0A / L0B; the matrix computation unit reads data from L0A / L0B to perform matrix computation; the computation result is then moved to L0C, and finally transported back to global memory 200 or the L1 buffer via a fixed pipe. Therefore, the data flow for matrix computation can be represented as GM-L1-L0A / L0B-[Cube]-L0C-FixPipe-GM, or GM-L1-L0A / L0B-[Cube]-L0C-FixPipe-L1.

[0089] It should be noted that the AI ​​Core 100 may also include a control unit ( Figure 1(Not shown in the diagram). The control unit includes at least one of the following: System Control, Instruction Dispatch, Cube Queue, Vector Queue, and MTE Queue. The System Control module is responsible for directing and coordinating the overall operation mode of AI Core 100, configuring parameters, and implementing power consumption control. When instructions are sequentially issued through the Instruction Dispatch module, they will be sent to the Cube Queue, Vector Queue, and MTE Queue respectively, depending on the type of instruction. In this way, the Cube computation unit, Vector computation unit, and MTE engine can execute corresponding tasks according to the instructions in their respective queues.

[0090] Currently, the implementation of scatter operators on computing card 10 involves calculating the amount of data each AI core 100 needs to process based on the amount of data to be processed. Then, based on the amount of data to be processed by each AI core, a portion of the index and its corresponding value are moved from global memory 200 into a unified buffer UB. The computing unit reads the value from the UB according to the index and uses the `atomic_add` instruction to accumulate the elements in the input X corresponding to the value in global memory 200. For ease of understanding, this application provides an example. For example... Figure 2 As shown, the index includes {3,0,5,3}, where the index indicates the position where data processing is required, for example, the rows with indexes 3, 0, 5, and 3 in the input data X. MTE2 moves a portion of the index and its corresponding value into UB, for example, moving a portion of the index {3,0,5} and its corresponding value into UB. MTE3 moves the values ​​corresponding to the aforementioned partial indices to global memory 200 and processes them with the elements in input X corresponding to the indices, for example, by accumulating them to obtain the calculation result.

[0091] In the above scheme, the processing order of multi-core processing cannot be guaranteed. In scenarios where the input data is floating-point numbers, the order of accumulation of floating-point numbers may affect the precision of the final calculation result. In scenarios requiring precise results or repeatability, "deterministic computation" is usually required. "Deterministic computation" refers to a calculation method where, under the same initial conditions, the calculation result is always the same regardless of how many times the calculation process is repeated. In other words, given the same input data and computing environment (such as the same hardware), deterministic computation will produce completely consistent outputs. Taking financial transaction systems as an example, operations such as interest calculation and transaction settlement require deterministic computation to ensure that the calculation results of repeated calculations are consistent and to avoid chaos in financial transactions. Based on this, in some cases, such as when the input data is floating-point numbers and deterministic computation is enabled, the above scheme will degenerate into using a single core for calculation, resulting in a sharp increase in end-to-end time and severe end-to-end performance degradation.

[0092] In view of this, this application provides an operator acceleration method. This method can be applied to a computing card. The computing card includes global memory and multiple AI cores. Specifically, the computing card can acquire a computational task to be executed. The task information of the computational task includes input data for a scattering operator, at least one source data, and index data corresponding to at least one source data. The input data is the data to be processed using the source data, and the index data indicates the position of the data to be processed within the input data. The computing card can determine the target core corresponding to each index in the index data from among the multiple AI cores. Indexes indicating the same position in the index data correspond to the same target core. The computing card can move the source data corresponding to each index in the index data into the target core corresponding to each index. Based on the source data and input data moved into the target core, a first computational task is executed within the target core to obtain an intermediate result. This intermediate result can be moved into global memory. Then, based on the intermediate result, a second computational task is executed between the target cores to obtain the operator execution result.

[0093] This method introduces a core-sharing scheme where the same index corresponds to the same core. This ensures that source data corresponding to indexes at the same location are preferentially moved to the same AI core for computation. Thus, even if the input data is floating-point or the computation task is deterministic, it does not need to degenerate into using a single core. It can achieve multi-core concurrent processing while maintaining accuracy, improving multi-core parallelism, shortening end-to-end latency, and enhancing end-to-end performance. Furthermore, this method merges index data, meaning that intermediate results obtained by moving indexes at the same location and their corresponding source data to the same AI core only need to be moved to global memory once. For scenarios with a large number of duplicate indexes, this method can significantly reduce MTE3 data transfer time and effectively improve the bandwidth utilization of computing cards (such as SIMD chips).

[0094] To make the technical solution of this application clearer and easier to understand, the operator acceleration method of this application will be described below with reference to the accompanying drawings.

[0095] See Figure 3 The flowchart shown is an operator acceleration method, which includes the following steps:

[0096] S302, The computing card obtains the computing tasks to be executed.

[0097] A computational task refers to a task performed to execute computational operations. Computational operations can be characterized by operators. In this application, the computational operation indicated by the computational task can be a scattering operation, which is characterized by scattering operators. Scattering operators distribute data from one set (or source set) to another set (or target set) according to rules. Different operations can be performed when distributing data to another set. Some scattering operators are used to perform update operations when distributing data to another set. For example, the `indexput` operator is used to perform update operations when distributing data to another set. Some scattering operators are used to perform accumulation operations when distributing data to another set. For example, the `scatteradd` operator and the `indexadd` operator are used to perform accumulation operations when distributing data to another set.

[0098] The task information for the computational task includes the input data for the scattering operator, at least one source data set, and index data corresponding to at least one source data set. The input data is the data to be processed using the source data. The index data indicates the location of the data to be processed within the input data. In other words, the index data indicates which data in the input data to perform the corresponding operation. It should be noted that the scattering operator can scatter along a specified dimension. For example, when the input data is a second-order tensor, it can be specified to scatter along the 0th dimension (e.g., rows) or the 1st dimension (e.g., columns). Assuming the input data has a shape of (3,5), index data of [0,1,4], and the specified dimension as the 1st dimension, this means performing the corresponding operation on the columns with indices (number, rank) of 0, 1, and 4 in the input data.

[0099] The source data can be data from a source set. The input data can be data from a target set. The source data can be a tensor or a scalar, and the input data can be a tensor. When the source data is a scalar, it means performing the same operation on the data at a specified position in the target set. For example, if the source data is 0.5 and the operation type is "add", it means adding 0.5 to the data at the specified position in the target set (the specified data in the input data). When the source data is a tensor, such as a source data tensor formed by multiple source data, it means performing the corresponding operation on the data at the specified position in the target set according to each element of the tensor. For example, if the source data is [0.1, 0.2, 0.3] and the operation type is "put", it means performing an update operation on the data at the specified position in the target set according to each element of the tensor.

[0100] In practical implementation, the computing card can receive a computing graph or a subgraph obtained by splitting the computing graph from the host of the computing device, and generate at least one computing task based on the computing graph or subgraph. These computing tasks may include, but are not limited to, matrix computing tasks or vector computing tasks. These computing tasks can be added to corresponding queues for execution. Specifically, matrix computing tasks can be added to a matrix operation queue, and vector computing tasks can be added to a vector operation queue. The computing card can retrieve computing tasks from the queues; for example, it can retrieve computing tasks for executing scatter operators from the vector operation queue.

[0101] S304. The calculation card transforms the index data in the task information to obtain the transformed index data.

[0102] Specifically, the computing card can preprocess the index data, converting it into a one-dimensional positive index for easier sorting. The preprocessing process is described in detail below.

[0103] In some possible implementations, the indices in the index data can be negative or positive. The computing card can transform all the indices in the index data so that each index in the transformed index data is positive. Alternatively, the computing card can transform the negative indices in the index data so that each index in the transformed index data is positive.

[0104] In other possible implementations, the index data can include multiple sets of indices. In deep learning, tensors can be viewed as high-dimensional arrays with multiple indices. Taking Graph Convolutional Networks (GCNs) as an example, the input tensor (input data in tensor form) has indices such as height, width, and number of channels. When performing a convolution operation with the kernel tensor, the computation card can align multiple sets of indices. The indices of the convolution kernel and the indices of the input tensor are aligned appropriately in both the spatial dimensions (height and width) and the channel dimensions, ensuring that the convolution operation correctly extracts image features. Based on this, the computation card aligns multiple sets of indices in the index data to obtain a one-dimensional index. For example, the computation card can invoke a deep learning framework to align multiple sets of indices to obtain a one-dimensional index.

[0105] The computing card obtains a one-dimensional positive index by performing positive-to-negative conversion and / or axis merging on the index data. It should be noted that step S304 above is optional, and the operator acceleration method of this application may not include this step. For example, if all indices in the index data are positive and only one set of indices is included, then step S304 above can be omitted.

[0106] S306. The calculation card sorts the converted index data to obtain an ordered index set.

[0107] Specifically, the computation card can count the occurrences of different indices in the transformed one-dimensional positive indexes, and then sort the indices according to their occurrence counts to obtain an ordered index set. The computation card can iterate through the transformed one-dimensional positive indexes, remove duplicates, and obtain the occurrence count of each index in the deduplicated index data. It should be noted that if the computation card does not execute S304, it can sort the index data to obtain an ordered index set. For example, the computation card can remove duplicates from the index data to obtain the occurrence count of each index in the deduplicated index data.

[0108] When sorting indexes, the computation card can sort them in descending order of frequency of occurrence. For example, the computation card can sort different indexes in descending order of their frequency of occurrence to obtain an ordered set of indexes. The indexes in the ordered set can be unique ordered indexes. For example, when sorting in descending order, the indexes in the ordered set can be unique indexes sorted by their frequency of occurrence from highest to lowest.

[0109] S308. The calculation card processes the input data using the source data based on the ordered index set to obtain the operator execution result.

[0110] The computation card can move the source data corresponding to each index in the ordered index data into the target kernel corresponding to each index for processing. For any index in the index data, the target kernel refers to the AI ​​kernel responsible for processing that index. The computation card uses the source data corresponding to the index in the target kernel to process the data at the index-indicated position in the input data. The processing method can be determined according to the operation type of the scattering operator. For example, if the operation type of the scattering operator is "add", an accumulation operation can be performed. This accumulation operation can be performed on the same index, adding the source data corresponding to that index to the data at the index-indicated position in the input data. As another example, if the operation type of the scattering operator is "put", an update operation can be performed. This update operation can be performed on the source data corresponding to the index, updating the data at the index-indicated position in the input data. For duplicate indexes, the update operation can be performed on one of the source data corresponding to the duplicate index, updating the data at the index-indicated position in the input data. For example, the update operation can update the latest source data (e.g., the source data corresponding to the last appearing index in the duplicate index) or the oldest source data (e.g., the source data corresponding to the earliest appearing index in the duplicate index) in the source data corresponding to the index to the data at the index-indicated position in the input data.

[0111] In this application, in order to achieve multi-core parallelism, the computing card can adopt a core-sharing strategy of same index and same core, and determine the same target core for indexes indicating the same position.

[0112] In some possible implementations, the computing card may disregard the data type of the input data and the type of computation task, employing a core-sharing strategy with the same index and core for different data types or computation types. In this case, indices indicating the same position in the index data correspond to the same target core. Data types can include integer types (or simply integers) and floating-point types (or simply floating-point types). Floating-point types can be further subdivided into different precision floating-point types, such as half-precision floating-point (typically identified by 16 bits / 2 bytes, hence also called FP16) or 16-bit brain float 16 (BF16), single-precision floating-point (such as FP32), or double-precision floating-point (FP64). Computation types can include deterministic computation and non-deterministic computation. Non-deterministic computation can be either deterministic computation disabled or not enabled.

[0113] In other possible implementations, the computing card can automatically select a core-sharing strategy to allocate cores to the indexes in the index data based on the data type of the input data, thereby improving the execution performance of operators (e.g., operator kernels). For example, when the data type of the input data is a target floating-point type, such as FP16 or BF16, the core-sharing strategy can be "same index, same core," meaning that for indices indicating the same position in the index data, the computing card can determine the same target core from multiple AI cores. The computing card can also automatically select a core-sharing strategy to allocate cores to the indexes in the index data based on the task type of the computation task. For example, when the task type of the computation task is deterministic computation, the core-sharing strategy can be "same index, same core," meaning that for indices indicating the same position in the index data, the computing card can determine the same target core from multiple AI cores. Furthermore, when the data type of the input data is not a target floating-point type and the computation type of the computation task is deterministic computation, the computing card can determine the same target core from multiple AI cores for indices indicating the same position in the index data.

[0114] In some cases, the computing card can also determine the target core using other core-sharing strategies. These strategies can include an index-based even distribution strategy or a random allocation strategy. Specifically, an index-based even distribution strategy means uniformly distributing the indices in the index data across multiple AI cores. When the input data is not of the target floating-point type and the computation type is non-deterministic, or when the input data is of a non-floating-point type (e.g., integer), the core-sharing strategy can be either an index-based even distribution strategy or a random allocation strategy. Accordingly, the computing card can use either an index-based even distribution strategy or a random allocation strategy to determine the target core corresponding to each index in the index data.

[0115] Furthermore, during data processing within the target core, considering accuracy issues, if the precision of the input data's floating-point type does not meet the task objective, the computing card can convert the input data's floating-point type from a first floating-point type to a second floating-point type, obtaining the converted input data. The second floating-point type has higher precision than the first. For example, the first floating-point type can be FP16 or BP16, and the second floating-point type can be FP32. Accordingly, the computing card can execute the first computation task within the target core based on the source data loaded into the target core and the converted input data, obtaining intermediate results. These intermediate results are then loaded into global memory, and a second computation task is executed between the target cores based on these intermediate results to obtain the operator execution result. By converting the input data to a high-precision second floating-point type, the accuracy of the operator execution result can be guaranteed.

[0116] To facilitate understanding, the following section will explain in detail the determination of the target core and the intra-core and inter-core processing of the target core, based on the selection strategy of the core distribution strategy according to the data type of the input data or the computation type of the computation task.

[0117] The computation card processes the input data using the source data based on the ordered index set, which may include the following steps:

[0118] S3081. The calculation card determines whether the input data is a floating-point type. If not, proceed to S3082 to S3085; if yes, proceed to S3086.

[0119] S3082. The computing card uses an index distribution strategy or a random allocation strategy to determine the target core corresponding to each index in the index data.

[0120] The index distribution strategy refers to evenly allocating indexes across multiple AI cores. In one implementation, the computing card can determine the number of indexes processed by each AI core based on the total number of indexes in the index data and the number of AI cores. Specifically, when the total number of indexes in the index data is an integer multiple of the number of AI cores, the number of indexes processed by each AI core can be the ratio of the total number of indexes to the number of AI cores. When the total number of indexes in the index data is not an integer multiple of the number of AI cores, the number of indexes processed by some AI cores can be the rounded-up or rounded-down result of the ratio of the total number of indexes to the number of AI cores. In this case, the number of indexes processed by each AI core can be different, and the sum of the number of indexes processed by multiple AI cores equals the total number of indexes.

[0121] It should be noted that when the number of indexes currently pending processing by an AI core is not zero, the computing card can determine the index data to be processed by each AI core based on the number of indexes in the index data, the number of AI cores, and the number of indexes currently pending processing by each AI core. This ensures that the load on each AI core is as balanced as possible.

[0122] The random allocation strategy refers to randomly assigning an index to multiple AI cores. Specifically, the computing card can construct a random function, the output of which can be any one of the multiple AI cores. For each index in the index data, the computing card executes the random function once, and the AI ​​core output by the random function can be the target core corresponding to the index.

[0123] S3083, the computing card moves the source data corresponding to each index in the index data into the target core corresponding to each index.

[0124] When the source data is a scalar, the computation card can move the source data into all target kernels. When the source data is a tensor, the computation card moves the source data corresponding to each index in the index data into the target kernel corresponding to each index. Specifically, the computation card can move the source data or the source data corresponding to each index into the unified buffer UB of each target kernel corresponding to that index. Furthermore, the computation card can also move the index into the corresponding target kernel so that the target kernel can perform the corresponding operation according to the index. It should be noted that when the scattering operator processes vectors, the target kernel can be the vector computation kernel AIV in the AI ​​kernel. Specifically, multiple vector computation kernels in the AI ​​kernel can have the same index corresponding to the same vector computation kernel.

[0125] S3084. The computing card performs the first computing task in the target core based on the source data and input data loaded into the target core, and obtains intermediate results.

[0126] In this application, each target kernel can be assigned one or more sets of indices (or identical indices) indicating the same position. Accordingly, the computing card can process the data at the index-indicated positions in the input data using source data corresponding to the same index within the target kernel, at a group-by-group granularity. This processing could include, for example, accumulation, thereby executing the first computational task and obtaining intermediate results. For instance, when the operation type of the scattering operator is `add`, the computing card can perform same-index accumulation within the target kernel to obtain intermediate results. Same-index accumulation can involve accumulating the source data corresponding to the same index to a specified position in the input data, or accumulating the source data corresponding to the same index with specified data in the input data.

[0127] S3085 The computing card moves the intermediate results into global memory, and performs a second computing task on the target cores based on the intermediate results to obtain the operator execution results.

[0128] Specifically, the computing card can move intermediate results into global memory to perform inter-core computation or inter-core communication on the intermediate results, thereby enabling the execution of a second computation task between target cores and obtaining the operator execution result.

[0129] S3086. The calculation card determines whether the input data is of the target floating-point type. If not, proceed to S3087; if yes, proceed to S3088 to S308c.

[0130] The target floating-point type can be a low-precision floating-point type, including but not limited to FP16 or BF16. The computing card can determine whether the input data is of the target floating-point type based on the declaration of the input data.

[0131] S3087. The calculation card determines whether the task type of the calculation task is deterministic calculation. If yes, execute S3088, S3089, S308b, and S308c; if no, execute S3082 to S3085.

[0132] Specifically, the computing card can read the configuration information of the computing task to determine whether the computing task is a deterministic computation or whether deterministic computation is enabled. The configuration information can be set by the user according to business requirements. When the business requires accurate results or repeatability, the user can configure the computing task type to deterministic computation or enable deterministic computation. In some possible implementations, deterministic computation can be disabled by default if the user does not configure it. In other possible implementations, deterministic computation can be enabled by default if the user does not configure it.

[0133] S3088, the computing card determines the target core corresponding to each index in the index data from multiple AI cores, indicating that the target cores corresponding to the same indexes are the same.

[0134] Specifically, the computing card employs a same-index, same-core strategy, thereby determining the target core corresponding to each index in the index data among multiple AI cores, ensuring that indices with the same indication position correspond to the same target core. The following is a detailed explanation of the same-index, same-core core-sharing strategy.

[0135] The computing card can determine the target core corresponding to each index in an ordered index set based on the amount of tasks to be processed by at least one AI core among multiple AI cores. The amount of tasks can be positively correlated with the number of indexes allocated to that AI core. Specifically, the computing card can determine the target core corresponding to each index in the ordered index set through load balancing based on the amount of tasks to be processed by at least one AI core among multiple AI cores. Specifically, if the indexes in the ordered index set are arranged in descending order of their occurrence count, the computing card can determine the target core corresponding to each index in the ordered index set sequentially. For the i-th index in the ordered index set, the computing card can select the AI ​​core with the smallest amount of tasks to be processed as the target core corresponding to that index. Then, the computing card can update the amount of tasks to be processed by the target core corresponding to that index based on the occurrence count of that index, and determine the target core corresponding to the (i+1)-th index based on the updated amount of tasks.

[0136] The above describes the specific implementation of determining the target core corresponding to each index in the deduplicated index data from multiple AI cores based on the frequency of each index in the deduplicated index data. In practical applications, other methods can also be used to determine the target core corresponding to each index from multiple AI cores.

[0137] S3089. The computing card moves the source data corresponding to each index in the index data into the target core corresponding to each index.

[0138] S308a When the precision of the floating-point type of the input data does not meet the task objective of the calculation task, the calculation card converts the floating-point type of the input data from the first floating-point type to the second floating-point type to obtain the converted input data.

[0139] Specifically, the input data is of type 1 floating-point. If the precision of the first floating-point type does not meet the task objective, the computing card can convert the input data type to improve its precision. The converted input data can be of type 2 floating-point, which has higher precision than the first floating-point type. In some examples, the first floating-point type can be FP16 or BF16, and the second floating-point type can be FP32 or FP64.

[0140] See Figure 4 The diagram illustrates a data type conversion. The input data is of type BF16 or FP16. The computing card can insert a type casting function outside the kernel of the scattering operator to be executed. In this example, the scattering operator can be `index_add`, and the type casting function can include, but is not limited to, `cast`. Accordingly, the computing card can call the kernel of the scattering operator to perform type conversion on the input data and execute the `index_add` operation on the converted input data. The converted input data can be of type FP32.

[0141] It should be noted that for scenarios where the floating-point type of the input data is BF16 or FP16, which does not meet the task precision, this method can specifically convert the input data involved in the calculation, further shortening the end-to-end time, and can also specifically move the data in, reducing redundant memory.

[0142] S308b: The computing card performs the first computing task in the target core based on the source data and (converted) input data loaded into the target core, and obtains intermediate results.

[0143] When the precision of the floating-point type of the input data does not meet the task objective, the computing card executes S308a to perform type conversion on the input data. The computing card can then perform the first computation task within the target core based on the source data loaded into the target core and the converted input data, obtaining an intermediate result. Conversely, when the precision of the floating-point type of the input data meets the task objective, the computing card may not execute S308a. Accordingly, the computing card can still perform the first computation task within the target core based on the source data loaded into the target core and the input data, obtaining an intermediate result.

[0144] The S308c computing card moves intermediate results into global memory and performs a second computing task on the target cores based on the intermediate results to obtain the operator execution results.

[0145] For the specific implementations of S3089, S308b, and S308c, please refer to the descriptions of S3083 to S3085, which will not be repeated here.

[0146] S310, The calculation card outputs the operator execution result.

[0147] Specifically, the computing card can output the operator execution results to the host, and further, the computing card can also return the operator execution results to the user through the host.

[0148] It should be noted that S310 is an optional step in the embodiments of this application, and the operator acceleration method of this application may not need to execute S310.

[0149] Based on the above description, it can be seen that the operator acceleration method of this application uses a core-sharing scheme with the same index on the same core. This allows source data corresponding to the same index to be moved to the same AI core for computation. Thus, even if the input data is a floating-point number and deterministic computation is enabled, it does not need to degenerate into using a single core for computation. It can achieve multi-core concurrent processing while maintaining accuracy, improving multi-core parallelism, shortening end-to-end latency, and enhancing end-to-end performance. Furthermore, this method merges index data, indicating that intermediate results obtained by moving indexes at the same position and their corresponding source data to the same AI core for computation can be moved to global memory only once. For scenarios with a large number of duplicate indexes, this method can significantly reduce MTE3 transport time and effectively improve the bandwidth utilization of the computing card.

[0150] This invention proposes a core-sharing optimization algorithm. This algorithm supports automatically selecting a reasonable and efficient core-sharing strategy from multiple strategies based on the data type of the input data or the computation type of the task, thereby improving the execution performance of scattering operators. The core-sharing strategies can include an even index distribution strategy, a random allocation strategy, and a same-index, same-core strategy. The specific implementations of operator acceleration using the even index distribution strategy and the same-index, same-core strategy are described in detail below.

[0151] First, see Figure 5 The diagram illustrates a process for operator acceleration based on an index-sharing strategy. The computing card (e.g., its control unit) can perform an index-sharing operation on the index data and allocate it to multiple AI cores. Each AI core allocated to an index is the target core corresponding to that index. Then, the source data corresponding to the index can be moved into global memory to the target core corresponding to that index. Figure 5This example uses the target core as one of multiple AI cores. Within an AI core, source data corresponding to the same index is accumulated. The AI ​​core can determine if the accumulation count has reached its limit, or if the remaining indexes are inconsistent with the index currently being accumulated (referred to as the accumulation index). If the accumulation count reaches its limit or the indexes are inconsistent, the AI ​​core can move the intermediate results into global memory. If the accumulation count has not reached its limit, or if there is an index among the remaining indexes that is consistent with the index currently being accumulated, the AI ​​core can continue moving the source data corresponding to that index. After moving the intermediate results into global memory, the target core can also determine if the index processing within the core has ended. If yes, the core processing flow can end; otherwise, global memory can continue moving the source data corresponding to the index.

[0152] Next, see Figure 6 The diagram shows a process for operator acceleration based on the same index and same core strategy, which includes the following steps;

[0153] Step 1: Calculate index k in the statistical index data of the card. i The number of times n i .

[0154] Step 2: Calculate the number of occurrences n using the card. i For index k i Sort the indexes in descending order to obtain an ordered index set I.

[0155] Step 3: Initialize the calculation card S = [0] 1×M This is used to record the amount of tasks to be processed by each AI core.

[0156] S represents the workload, which can be determined by the number of tasks and the number of indexes. M represents the number of AI cores.

[0157] Step 4: The calculation card can traverse the ordered index set and set index k. i Assign the AI ​​core with the fewest pending tasks to the AI ​​core, and then update the number of pending tasks for that AI core.

[0158] Specifically, the calculation card can determine id = min(S) and update S[id] += n. i .

[0159] Step 5: The AI ​​core in the computing card performs in-core indexing based on the assigned index.

[0160] The process of processing the kernel index can be found by referring to... Figure 5 The illustrated embodiments will not be described in detail here. In this application, each target core only processes the assigned index, ensuring that the tasks of each AI core are as balanced as possible.

[0161] Based on the aforementioned operator acceleration method, this application also provides an operator acceleration device 700. The operator acceleration device 700 is deployed on a computing card 10, which includes an AI core 100 and global memory 200. The operator acceleration device 700 includes:

[0162] The task acquisition module 702 is used to acquire a computation task to be executed. The task information of the computation task includes input data of a scattering operator, at least one source data, and index data corresponding to the at least one source data. The input data is the data to be processed using the source data, and the index data is used to indicate the position of the data to be processed in the input data.

[0163] Core partitioning module 704 is used to determine the target core corresponding to each index in the index data from the plurality of AI cores, wherein the target cores corresponding to indexes indicating the same position in the index data are the same;

[0164] The task execution module 706 is used to move the source data corresponding to each index in the index data into the target core corresponding to each index, and to perform a first calculation task in the target core according to the source data moved into the target core and the input data to obtain intermediate results;

[0165] The task execution module 706 is further configured to move the intermediate results into the global memory, execute a second computation task in the target cores based on the intermediate results, and obtain the operator execution result.

[0166] For example, the task acquisition module 702, the core sub-module 704, and the task execution module 706 described above can be implemented in hardware or in software.

[0167] When implemented in software, the task acquisition module 702, the core-sharing module 704, and the task execution module 706 can be applications running on the computing card, such as a computing engine. When implemented in hardware, the task acquisition module 702, the core-sharing module 704, and the task execution module 706 can include at least one hardware unit; for example, the task acquisition module 702 can be a vector operation queue.

[0168] In some possible implementations, the core-splitting module 704 is specifically used for:

[0169] For indexes indicating the same position in the index data, the same target core is determined from the plurality of AI cores.

[0170] In some possible implementations, the core-splitting module 704 is specifically used for:

[0171] When the computation type of the computation task is deterministic computation, for the indexes in the index data that indicate the same position, the same target core is determined from multiple AI cores.

[0172] In some possible implementations, the task execution module 706 is specifically used for:

[0173] When the precision of the floating-point type of the input data does not meet the task objective of the computation task, the floating-point type of the input data is converted from a first floating-point type to a second floating-point type to obtain the converted input data. The precision of the second floating-point type is higher than that of the first floating-point type.

[0174] A first computational task is performed within the target core based on the source data loaded into the target core and the transformed input data.

[0175] In some possible implementations, the core-splitting module 704 is specifically used for:

[0176] The index data is deduplicated, and the number of occurrences of each index in the deduplicated index data is obtained;

[0177] Based on the number of times each index appears in the deduplicated index data, the target core corresponding to each index in the deduplicated index data is determined from the plurality of AI cores.

[0178] In some possible implementations, the core-splitting module 704 is specifically used for:

[0179] Based on the frequency of occurrence of each index in the deduplicated index data, the deduplicated index data is sorted to obtain an ordered index set;

[0180] Based on the amount of tasks to be processed by at least one of the multiple AI cores, the target core corresponding to each index in the ordered index set is determined.

[0181] In some possible implementations, the core-splitting module 704 is specifically used for:

[0182] Based on the amount of tasks to be processed by at least one AI core among the multiple AI cores, the target core corresponding to each index in the ordered index set is determined through load balancing.

[0183] In some possible implementations, the computing card is a SIMD chip.

[0184] This application also provides a multi-core architecture computing card, which includes multiple AI cores and global memory. The AI ​​cores can adopt a coupled architecture or a separate architecture. In a coupled architecture, vector computation units and matrix computation units are coupled to one AI core. In a separate architecture, the vector computation units and matrix computation units are separated, forming vector computation cores and matrix computation cores. In a separate architecture, multiple AI cores can include multiple vector computation cores. It should be noted that multiple AI cores can share a matrix computation core. The following example illustrates a computing card with a separate architecture.

[0185] like Figure 1 As shown, the computing card 10 includes an AI core 100 and a global memory 200. The AI ​​core 100 includes a matrix computing core 102 and multiple vector computing cores 104. The global memory can store computer-readable instructions. When these computer-readable instructions are executed, the vector computing cores 104 execute the operator acceleration method of the aforementioned embodiment. Specifically, for the computing task to be executed, each index in the index data of the scattering operator to be executed corresponds to a vector computing core 104, wherein indices indicating the same position correspond to the same vector computing core 104. The source data corresponding to each index is loaded into the corresponding vector computing core 104. The vector computing core 104 can perform a first computing task within the core based on the loaded source data and input data to obtain intermediate results. Then, the intermediate results are loaded into the global memory, and a second computing task is performed between the cores based on the intermediate results to obtain the operator execution result.

[0186] Furthermore, this application also provides a computing cluster. The computing cluster includes multiple computing cards, which can be connected to each other via a bus or network. The architecture of the computing cluster is described below with reference to the accompanying drawings.

[0187] First, see Figure 8 The diagram illustrates a computing cluster architecture, which can be a single-machine, multi-GPU architecture. In this architecture, the computing cluster includes a CPU 11 on the host side and multiple computing cards 10 on the device side. Further, the host side may also include host memory 12. The CPU 11, host memory 12, and multiple computing cards 10 can be connected via a bus. In some examples, the CPU 11 can schedule a computation graph loaded into host memory 12 to computing cards 10, or it can partition the computation graph loaded into host memory 12 into subgraphs, and then schedule these subgraphs to computing cards 10. Computing cards 10 can run an operator acceleration device 700 to accelerate the execution of scatter-type operators in the computation graph or subgraphs.

[0188] The computing card 10 can acquire the computing task to be executed. The task information of the computing task includes the input data of the scatter operator, at least one source data, and the index data corresponding to at least one source data. It determines the target kernel corresponding to each index in the index data from multiple AI kernels. The target kernels corresponding to the same index in the index data are the same. Then, the source data corresponding to each index in the index data is moved into the target kernel corresponding to each index. The first computing task is executed in the target kernel according to the source data and input data moved into the target kernel to obtain intermediate results. Then, the intermediate results are moved into global memory 200. The second computing task is executed between the target kernels according to the intermediate results to obtain the operator execution result.

[0189] Secondly, see Figure 9 The diagram illustrates another computing cluster architecture. This computing cluster can be a multi-machine, multi-GPU architecture. In this architecture, the computing cluster can include multiple computing devices, each including a CPU 11 on the host side and multiple computing GPUs 10 on the device side. The host side may also include host memory 12. Computing GPUs 10 within the same computing device can be connected via a bus. Computing GPUs 10 in different computing devices can be connected via a network. The host-side CPU 11 can partition the computation graph into subgraphs and then schedule these subgraphs to the computing GPUs 10. Alternatively, the CPU 11 can directly schedule the computation graph to the computing GPUs 10. The computing GPUs 10 can accelerate the execution of scatter operators within the computation graph or subgraphs.

[0190] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing card or computing cluster to execute the aforementioned operator acceleration method. This application also provides a computer program product containing instructions. The computer program product can be software or program products containing instructions that can run on a computing device or be stored on any available medium. When the computer program product runs on a computing card or computing cluster, it causes the computing card or computing cluster to execute the aforementioned operator acceleration method.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An operator acceleration method, characterized by, Applied to a computing card, the computing card including multiple artificial intelligence (AI) cores and global memory, the method includes: Obtain the computation task to be executed. The task information of the computation task includes input data of a scattering operator, at least one source data, and index data corresponding to the at least one source data. The input data is the data to be processed using the source data, and the index data is used to indicate the position of the data to be processed in the input data. The target core corresponding to each index in the index data is determined from the plurality of AI cores, wherein the target cores corresponding to indexes indicating the same position in the index data are the same; The source data corresponding to each index in the index data is moved into the target core corresponding to each index. Based on the source data moved into the target core and the input data, the first calculation task is performed in the target core to obtain intermediate results. The intermediate results are moved into the global memory, and a second computational task is performed on the target cores based on the intermediate results to obtain the operator execution results.

2. The method of claim 1, wherein, The step of determining the target core corresponding to each index in the index data from the plurality of AI cores includes: For indexes indicating the same position in the index data, the same target core is determined from the plurality of AI cores.

3. The method of claim 2, wherein, Determining the target core corresponding to each index in the index data from the plurality of AI cores includes: When the computation type of the computation task is deterministic computation, for the indexes in the index data that indicate the same position, the same target core is determined from the multiple AI cores.

4. The method according to any one of claims 1 to 3, characterized in that, The step of performing a first computation task within the target core based on the source data loaded into the target core and the input data includes: When the precision of the floating-point type of the input data does not meet the task objective of the computation task, the floating-point type of the input data is converted from a first floating-point type to a second floating-point type to obtain the converted input data. The precision of the second floating-point type is higher than that of the first floating-point type. A first computational task is performed within the target core based on the source data loaded into the target core and the transformed input data.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the target core corresponding to each index in the index data from the plurality of AI cores includes: The index data is deduplicated, and the number of occurrences of each index in the deduplicated index data is obtained; Based on the number of times each index appears in the deduplicated index data, the target core corresponding to each index in the deduplicated index data is determined from the plurality of AI cores.

6. The method of claim 5, wherein, The step of determining the target core corresponding to each index in the deduplicated index data from the plurality of AI cores based on the frequency of occurrence of each index in the deduplicated index data includes: Based on the frequency of occurrence of each index in the deduplicated index data, the deduplicated index data is sorted to obtain an ordered index set; Based on the amount of tasks to be processed by at least one of the multiple AI cores, the target core corresponding to each index in the ordered index set is determined.

7. The method of claim 6, wherein, The step of determining the target core corresponding to each index in the ordered index set based on the amount of tasks to be processed by at least one AI core among the plurality of AI cores includes: Based on the amount of tasks to be processed by at least one AI core among the multiple AI cores, the target core corresponding to each index in the ordered index set is determined through load balancing.

8. The method according to any one of claims 1 to 7, characterized in that, The computing card is a Single Instruction Multiple Data (SIMD) chip.

9. An operator acceleration device, characterized in that, The device is deployed on a computing card, which includes multiple artificial intelligence (AI) cores and global memory. The device includes: The task acquisition module is used to acquire the computation task to be executed. The task information of the computation task includes input data of the scattering operator, at least one source data and index data corresponding to the at least one source data. The input data is the data to be processed using the source data, and the index data is used to indicate the position of the data to be processed in the input data. The core-splitting module is used to determine the target core corresponding to each index in the index data from the plurality of AI cores, wherein the target cores corresponding to indexes indicating the same position in the index data are the same; The task execution module is used to move the source data corresponding to each index in the index data into the target core corresponding to each index, and to perform a first calculation task in the target core according to the source data moved into the target core and the input data to obtain intermediate results; The task execution module is also used to move the intermediate results into the global memory, and execute a second computation task between the target cores based on the intermediate results to obtain the operator execution result.

10. The apparatus according to claim 9, characterized in that, The core-splitting module is specifically used for: For indexes indicating the same position in the index data, the same target core is determined from the plurality of AI cores.

11. The apparatus according to claim 10, characterized in that, The core-splitting module is specifically used for: When the computation type of the computation task is deterministic computation, for the indexes in the index data that indicate the same position, the same target core is determined from multiple AI cores.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The task execution module is specifically used for: When the precision of the floating-point type of the input data does not meet the task objective of the computation task, the floating-point type of the input data is converted from a first floating-point type to a second floating-point type to obtain the converted input data. The precision of the second floating-point type is higher than that of the first floating-point type. A first computational task is performed within the target core based on the source data loaded into the target core and the transformed input data.

13. The apparatus according to any one of claims 9 to 12, characterized in that, The core-splitting module is specifically used for: The index data is deduplicated, and the number of occurrences of each index in the deduplicated index data is obtained; Based on the number of times each index appears in the deduplicated index data, the target core corresponding to each index in the deduplicated index data is determined from the plurality of AI cores.

14. The apparatus according to claim 13, characterized in that, The core-splitting module is specifically used for: Based on the frequency of occurrence of each index in the deduplicated index data, the deduplicated index data is sorted to obtain an ordered index set; Based on the amount of tasks to be processed by at least one of the multiple AI cores, the target core corresponding to each index in the ordered index set is determined.

15. The apparatus according to claim 14, characterized in that, The core-splitting module is specifically used for: Based on the amount of tasks to be processed by at least one AI core among the multiple AI cores, the target core corresponding to each index in the ordered index set is determined through load balancing.

16. The apparatus according to any one of claims 9 to 15, characterized in that, The computing card is a Single Instruction Multiple Data (SIMD) chip.

17. A computing card, characterized in that, The computing card includes multiple artificial intelligence (AI) cores and global memory. The AI ​​cores are used to execute computer-readable instructions loaded into the global memory to perform the operator acceleration method as described in any one of claims 1 to 8.

18. A computing cluster, characterized in that, The computing cluster includes multiple computing cards as described in claim 17, which are connected to each other via a bus or network.

19. A computer-readable storage medium, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the operator acceleration method according to any one of claims 1 to 8.

20. A computer program product, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the operator acceleration method according to any one of claims 1 to 8.

Images