A hardware accelerator based on Softmax operation, a neural network processor and a chip

Abstract

Embodiments of the present application provide a hardware accelerator based on Softmax operation, a neural network processor and a chip. The steps of Softmax calculation are fused into a small amount of calculation tasks, which are executed by a data operation unit and a tensor engine. A tree-shaped comparator array and a repeat circuit are arranged in the data operation unit, a near-memory calculation unit is arranged at an on-chip memory port, a multi-path selector with switchable data paths and a second write data path are arranged in the tensor engine. Thus, the Softmax calculation in the related art which requires 4 to 5 independent jobs is reduced to a small amount of jobs, and the number of task scheduling and synchronization overhead is reduced. Meanwhile, the near-memory calculation unit completes the subtraction operation along with the process of writing the maximum value back to the memory, eliminating the write-back and read operation of the intermediate result of the maximum value, and reducing the memory bandwidth consumption. In addition, the multi-path selector enables the tensor engine to flexibly switch between different modes, while maximizing the reuse of existing hardware resources.

Description

Technical Field

[0001] This application relates to the field of hardware acceleration technology for neural network model computation, and more specifically, to a hardware accelerator, neural network processor and chip based on Softmax operation. Background Technology

[0002] Softmax is a widely used operator in deep neural networks, especially playing a key role in the attention mechanism of the Transformer architecture. Its mathematical expression is: The calculation includes multiple sub-steps such as finding the maximum value, exponentiation, summation and normalization. There are strict data dependencies between the sub-steps, that is, the calculation of the next step must wait for the result of the previous step.

[0003] In relevant NPU hardware architectures, Softmax computation is typically broken down into 4-5 independent computation tasks (Jobs). Each Job is completed sequentially by different execution units. The specific process is as follows: Job0 calculates the maximum value, Job1 calculates the exponent offset, Job2 calculates the exponent sum, Job3 calculates the reciprocal, and Job4 performs normalized multiplication.

[0004] The related technologies have the following drawbacks: a large number of jobs, resulting in high scheduling overhead, with each job's startup, scheduling, and synchronization consuming control resources; repeated data transfer, with intermediate results being passed between jobs via memory read and write, consuming a large amount of memory bandwidth; low utilization of execution units, with each execution unit idle while waiting for the preceding job to complete; and high overall computational latency, affecting the inference performance of models such as Transformer. Summary of the Invention

[0005] The purpose of this application is to provide a hardware accelerator, neural network processor and chip based on Softmax operation, which aims to solve the problems of large number of Softmax calculation jobs, large memory access overhead and low execution unit utilization in related technologies, so as to achieve the technical effect of reducing task scheduling and memory access overhead and improving computing efficiency.

[0006] The first aspect of this application provides a hardware accelerator based on Softmax operation, the hardware accelerator comprising: The execution unit pipeline includes a data processing unit and a tensor engine. The data processing unit is used to execute a first computation task, and the tensor engine is used to execute a second computation task. The first computation task and the second computation task together cover the maximum value operation sub-step, the exponentiation operation sub-step, the summation operation sub-step, the reciprocal operation sub-step, and the normalization operation sub-step. The data processing unit includes a tree comparator array and a repeat circuit. The tree comparator array is used to perform parallel comparisons on each row of the input data to obtain the maximum value of each row. The repeat circuit is used to copy the maximum value of each row to the same shape as the input data and write it back to the on-chip memory. A proximity calculation unit is provided at the port of the on-chip memory. The proximity calculation unit is used to read the input data from the on-chip memory at full bandwidth of the memory interface and perform a subtraction operation to obtain the difference data during the process of writing the maximum value of each row back to the on-chip memory, and write the difference data back to the on-chip memory. The tensor engine includes a lookup unit, a reduction unit, a first write data path, a second write data path, and a multiplexer. The multiplexer is used to switch the data path in Softmax mode as follows: the input data is sequentially processed by the lookup unit for exponent lookup, one path outputs the exponent value through the first write data path, and the other path is processed by the reduction unit for summation and then outputs the exponent sum through the second write data path.

[0007] In the above implementation, all sub-steps of Softmax computation are merged into a small number of computation tasks, which are executed separately by the data processing unit and the tensor engine. A tree comparator array and repeat circuit are set in the data processing unit, a near-memory computation unit is set at the on-chip memory port, and a switchable data path multiplexer and a second write data path are set in the tensor engine. This reduces the Softmax computation, which requires 4 to 5 independent jobs in related technologies, to a small number of jobs, reducing task scheduling and synchronization overhead. Simultaneously, the near-memory computation unit performs subtraction operations along the path during the maximum value write-back to memory process, eliminating the write-back and read-back operation of intermediate maximum values ​​and reducing memory bandwidth consumption. Furthermore, the multiplexer allows the tensor engine to flexibly switch between different modes, maximizing the reuse of existing hardware resources while accelerating Softmax and avoiding the area overhead caused by adding a large number of dedicated computation units for Softmax.

[0008] Furthermore, when the data processing unit processes the maximum value calculation and subtraction operation of the (i+1)th row of data, the tensor engine simultaneously processes the exponent lookup table and reduction summation operation of the i-th row of data; i is a non-negative integer, and i+1 is less than the total number of rows.

[0009] In the above implementation process, by limiting the data processing unit to process the data of the i+1th row while the tensor engine processes the data of the ith row, a two-stage pipeline parallel scheme is constructed, which enables the two execution units to overlap in execution on different rows of data, hides the waiting time caused by data dependency, and improves hardware utilization and computing throughput.

[0010] Furthermore, the tree comparator array is used to group the data elements of each row according to a preset number. Within each group, the maximum value of each group is obtained through multiple levels of pairwise comparisons. Then, the maximum values ​​of each group are compared through a single level of comparison to obtain a scalar maximum value for each row.

[0011] In the above implementation process, by grouping each row of data elements according to a preset number, obtaining the maximum value of each group through multi-level pairwise comparisons, and then obtaining the maximum value of the row through a single-level comparison, the time complexity of obtaining the maximum value is reduced, and a balance is achieved between hardware area and computing speed.

[0012] Furthermore, the on-chip memory is an OCM on-chip memory, the near-memory computing unit is located at the data port of the OCM on-chip memory, and the subtractor is located near the OCM on-chip memory.

[0013] In the above implementation, by placing the near-memory computing unit at the data port of the OCM on-chip memory and the subtractor physically near the OCM, the entire process of reading the original data, subtracting the maximum value, and writing back the difference is completed near the memory port with full interface bandwidth. This does not occupy the core computing path of the data processing unit. It avoids the round-trip overhead of transporting the intermediate result of the maximum value over a long distance through the on-chip network to the remote execution unit and then writing it back. It also allows the data processing unit to start processing the next row of data immediately after writing the maximum value back to the OCM, and to execute it in parallel with the subtraction operation. This maximizes the release of memory bandwidth and eliminates the waiting time of the execution unit.

[0014] Furthermore, the multiplexer is located at the input of the lookup table unit, and the control terminal of the multiplexer is connected to the Job configuration register. The multiplexer includes a first input, a second input, and an output. The first input is connected to the data input port of the tensor engine, the second input is connected to the output of the reduction unit, and the output of the multiplexer is connected to the input of the lookup table unit. When the control bit is in the first state, the multiplexer selects the first input terminal to connect to the input terminal of the lookup unit, so that the input data is sent to the lookup unit for exponential lookup, the lookup result is output through the first write data path, and the output of the lookup unit is sent to the reduction unit for summation and then output through the second write data path. When the control bit is in the second state, the multiplexer selects the second input terminal to connect to the input terminal of the lookup unit, so that the input data is processed by the reduction unit and then sent to the lookup unit for lookup, and the lookup result is output through the first write data path.

[0015] In the above implementation process, by setting the multiplexer at the input of the lookup unit and using a single control bit in the Job configuration register to switch between the two data paths, the tensor engine can achieve the fusion of exponential lookup and reduction summation in a single job by adding only a multiplexer and a write data path. At the same time, it retains TENG's full support for general tensor operations, and gains the dual capabilities of Softmax's dedicated computing and general computing with a very small increase in hardware area.

[0016] Further, the first calculation task includes: the data processing unit calculates the maximum value of each row of data, copies it to the same shape as the input data through the repeat circuit, and writes it back to the on-chip memory; at the same time, the near-memory calculation unit is triggered to perform a subtraction operation to obtain the difference data and write it back to the on-chip memory; the difference data is used as the output of the first calculation task. The second computation task includes: the tensor engine reads the difference data from the on-chip memory, performs exponent lookup and reduction summation operations in the Softmax mode, and outputs the exponent value and the exponent sum.

[0017] In the above implementation process, the boundaries of the sub-steps completed by each of the two computing tasks and the output data flow direction are defined, so that the scheduler can complete the issuance of the two jobs with the least amount of information. At the same time, it ensures that there is a deterministic data flow connection between the storage location of the difference data output by the first computing task in the on-chip memory and the read address of the second computing task, thus providing timing guarantees for pipeline parallelism.

[0018] Furthermore, the hardware accelerator also includes: The reciprocal and normalization execution unit is used to calculate the reciprocal of the exponent and the sum, and multiply the exponent value by the reciprocal to obtain the Softmax normalization result.

[0019] In the above implementation process, by setting up independent execution units for reciprocal calculation and normalization, the two sub-steps of reciprocal calculation and normalization multiplication in Softmax calculation are separated from the tensor engine and completed by dedicated hardware. This allows the tensor engine to start processing the next batch of data after completing the exponent lookup table and reduction summation output, without waiting for the reciprocal calculation and normalization operations to complete.

[0020] Furthermore, the shape of the input data is (M, N); where M is the batch processing dimension and N is the sequence length dimension, and the N data elements in each row are divided into multiple batches and input into the tree comparator array.

[0021] In the above implementation process, each batch of data enters the comparator array sequentially for intra-group reduction, and the maximum value of each group is then compared at the final stage to obtain the row maximum value. When N is much larger than the preset number of groups, this batch processing method enables the limited comparator hardware resources to process sequences of arbitrary lengths, can adapt to different sequence length configurations in Transformer, and has good scalability.

[0022] A second aspect of this application provides a neural network processor, including any of the hardware accelerators described in the first aspect.

[0023] A third aspect of this application provides a chip, on which a hardware accelerator as described in any of the first aspects or a neural network processor as described in the second aspect is integrated. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 A schematic diagram of a DAU hardware microarchitecture provided in this application embodiment; Figure 2 This is a schematic diagram of a TENG hardware microarchitecture provided for an embodiment of this application. Detailed Implementation

[0026] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0027] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0028] To address any of the problems raised above, this application provides a hardware accelerator based on Softmax operation. In this embodiment, the hardware accelerator includes: The execution unit pipeline includes a data processing unit and a tensor engine. The data processing unit is used to execute a first computation task, and the tensor engine is used to execute a second computation task. The first computation task and the second computation task together cover the maximum value operation sub-step, the exponentiation operation sub-step, the summation operation sub-step, the reciprocal operation sub-step, and the normalization operation sub-step. The data processing unit includes a tree comparator array and a repeat circuit. The tree comparator array is used to perform parallel comparisons on each row of the input data to obtain the maximum value of each row. The repeat circuit is used to copy the maximum value of each row to the same shape as the input data and write it back to the on-chip memory. A proximity calculation unit is provided at the port of the on-chip memory. The proximity calculation unit is used to read the input data from the on-chip memory at full bandwidth of the memory interface and perform a subtraction operation to obtain the difference data during the process of writing the maximum value of each row back to the on-chip memory, and write the difference data back to the on-chip memory. The tensor engine includes a lookup unit, a reduction unit, a first write data path, a second write data path, and a multiplexer. The multiplexer is used to switch the data path in Softmax mode as follows: the input data is sequentially processed by the lookup unit for exponent lookup, one path outputs the exponent value through the first write data path, and the other path is processed by the reduction unit for summation and then outputs the exponent sum through the second write data path.

[0029] It should be noted that an execution unit pipeline refers to arranging multiple execution units in series, with each execution unit undertaking a step in the computation process. Data flows directly into the next execution unit after being processed by the previous one. In this embodiment, the execution unit pipeline includes a data processing unit and a tensor engine.

[0030] The advantage of an execution unit pipeline is that once the pipeline is full, each cycle can output a processing result, rather than waiting for the entire task to complete before processing the next batch of data. For input data of shape (M,N), when M is large, the pipeline throughput approaches the processing rate of a single execution unit, and the overall speedup is close to the pipeline stage number.

[0031] In general-purpose NPU architectures, the Data Arithmetic Unit (DAU) is originally responsible for data preprocessing, including comparisons and simple arithmetic operations. This application involves targeted hardware modifications to the DAU to enable it to efficiently perform maximum value calculation and subtraction preprocessing in Softmax computation. In this application, the DAU processes input data of shape (M, N) line by line, outputting the maximum value for each line. This maximum value is then copied and expanded before participating in the subtraction operation, ultimately outputting the difference data.

[0032] In general-purpose NPU architectures, the Tensor Engine (TENG) is responsible for complex tensor-level operations, including multiplication, lookup table (LUT) operations, and reduce operations. This application modifies the TENG's data path, enabling it to sequentially perform exponent lookup and reduce summation within a single job. TENG reads the difference data from the DAU output from on-chip memory, performs an exponent lookup for each element, and simultaneously performs reduce summation on the exponent values ​​of each row, outputting the exponent value and the exponent sum.

[0033] The first and second computational tasks are two independently scheduled jobs that work together to complete the Softmax computation. The first computational task (executed by DAU) covers the maximum value sub-step; the second computational task (executed by TENG) covers the exponentiation and summation sub-steps; the reciprocal and normalization sub-steps can be completed by additional execution units or internally by TENG through a reciprocal table lookup. This partitioning reduces the scheduling of 4-5 jobs in related technologies to 2 jobs, significantly reducing scheduling overhead.

[0034] The first and second computational tasks are treated as a single unit at the scheduling level. The scheduler only needs to issue the job configuration twice, namely the DAU Job and the TENG Job, instead of the 4-5 times required in related technologies. In some embodiments, pipeline overlap can exist between the first and second computational tasks, which further improves execution efficiency.

[0035] The tree comparator array is a new hardware module added within the DAU in this embodiment. It is used to efficiently complete the maximum value calculation sub-step in the Softmax calculation. Its hardware structure and working principle are described below: The tree-structured comparator array consists of multiple levels of comparators connected in a tree topology. Each comparator receives two input data elements, compares them, and outputs the larger value. The first-level comparator receives data elements paired together and outputs a candidate maximum value; the second-level comparator receives the output of the first level and continues to compare them pairwise; and so on, until the root node outputs the maximum value among all data elements.

[0036] Taking a group of 16 data elements as an example, the first level of 8 comparators compares each pairwise to obtain 8 candidate values, the second level of 4 comparators obtains 4 candidate values, the third level of 2 comparators obtains 2 candidate values, and the fourth level of 1 comparator obtains the maximum value of the group. This tree-structured reduction reduces the latency of the comparison operation from O(N) to O(log₂N), significantly reducing the number of clock cycles required to find the maximum value.

[0037] The repeat circuit is another hardware module added inside the DAU in this embodiment. It is used to expand the maximum value to the same shape as the original input data. The shape of the maximum value output by the tree comparator array is (M, 1), that is, there is only one scalar per row. Subtraction operation requires subtracting the maximum value corresponding to the row from each original data element, requiring the shape of the maximum value data to be the same as the original input data, that is, (M, N). The function of the repeat circuit is to copy the one maximum value of each row N times to form a maximum value tensor with shape (M, N). In hardware implementation, the repeat circuit can be implemented through data broadcast logic, that is, after the maximum value of each row is latched, it is continuously output N times within N clock cycles to complete the copying to N positions.

[0038] For example, after receiving input data of shape (M,N), the DAU, for the i-th row (i=0,1,...,M) 1) A tree comparator array compares the N elements of the row in parallel to obtain the scalar maximum value max(x). i Subsequently, the repeat circuit copies the maximum value N times, forming a maximum value vector of shape (1, N); this vector is written back to the on-chip memory through the write data path. During this process, the near-memory computing unit performs the subtraction operation along the path.

[0039] Near-Memory Computing (NMC) units enable data access from nearby locations and eliminate repeated transfer of intermediate results. NMC places computational logic physically close to the memory, allowing computation to be completed while data is being read from memory, thus avoiding long-distance data transfer between memory and distant computation units. Optionally, the NMC is located at a port of the On-Chip Memory (OCM) and performs subtraction operations along the way during the maximum value write-back to the OCM. Specifically, the NMC reads the original input data from the OCM, simultaneously receives the maximum value written by the Data Active User (DAU), performs the subtraction operation, and immediately writes the difference back to the OCM. The entire read-subtract-write process is completed near the memory port, resulting in short data transfer distances.

[0040] The advantage of near-memory computing is that it completes calculations at the upper limit of the memory interface bandwidth, without occupying the computation cycle of the execution unit. Specifically: First, it avoids the overhead of transporting intermediate results. In related technologies, the maximum value result needs to be written to the OCM via the write data path of the memory, and then the exponential offset Job reads the maximum value from the OCM via the read data path. This process of writing and reading wastes twice the memory bandwidth. In this embodiment, however, the maximum value is used for subtraction by the near-memory computing unit at the same time as it is written to the OCM, eliminating the need for additional read operations. Secondly, it does not occupy the core computing path of DAU. The subtraction operation is completed by the near-memory computing unit at the OCM port. After the DAU writes the maximum value back to the OCM, it can start processing the next row of data without waiting for the subtraction to complete. This achieves decoupling and parallelism between the maximum value retrieval and the subtraction operation. Third, full bandwidth utilization. The read / write speed of the near-memory computing unit (OCM) is consistent with the physical bandwidth of the memory interface, so there is no bandwidth bottleneck.

[0041] The lookup table unit is a hardware module within TENG used to perform nonlinear function calculations. It internally stores a table of approximate values ​​for exponential functions. Input data serves as the index address, and the lookup table unit outputs the corresponding function value. In Softmax calculations, the lookup table unit is used to perform the exponential operation exp(x) i max(x i The input is the difference data from the DAU output, and the output is the corresponding exponent value. Using a lookup table to perform exponentiation has higher hardware efficiency and lower latency compared to direct calculation.

[0042] The reduction unit is a hardware module within TENG used to implement tensor reduction operations. In Softmax computation, the reduction unit is used to perform the summation operation ∑exp(x) of the exponent values. j max(x j For example, the hardware implementation of the reduction unit is an adder tree or an accumulator. The reduction unit receives the exponent values ​​output by the lookup unit, and adds the N exponent values ​​of the same row element by element to obtain the exponent sum of that row. After reduction, the exponent sum is output through the second write data path. The reduction unit and the adder reuse the same hardware resources. In TENG's original mode, the reduction unit outputs the result to the lookup unit after reduction; in Softmax mode, the lookup unit outputs the result to the reduction unit. This hardware reuse design extends the functionality of TENG without increasing the area.

[0043] The first write data path (WDMA0) is an inherent write data path of TENG, used to write the output (exponent value) of the lookup unit to the on-chip memory. The second write data path (WDMA1) is a new write data path added within TENG in this embodiment, used to independently write the output (exponent sum) of the reduction unit to the on-chip memory. The reason for adding the second write data path in this embodiment is that in Softmax mode, the output of the lookup unit needs to be output in two copies simultaneously: one copy is written out as the exponent value through the first write data path, and the other copy is sent to the reduction unit for summation and then written out through the second write data path. Without the second write data path, the output of the reduction unit can only occupy the first write data path, resulting in the exponent values ​​not being able to be output simultaneously, or requiring additional scheduling delays.

[0044] The multiplexer (MUX) is a new hardware module added to the TENG in this embodiment, used to switch the data path between Softmax mode and the original mode. The multiplexer works as follows: it has multiple data inputs and one data output. A selection control signal connects one of the data inputs to the data output. In this embodiment, the first input of the multiplexer is connected to the output of the lookup unit, receiving data before reduction processing; the second input is connected to the output of the reduction unit, receiving data after lookup processing. The output of the multiplexer is connected to the input of the first write data path. By switching the selection state of the multiplexer, the TENG's data path can switch between the two modes, enabling TENG to simultaneously support general-purpose computing (i.e., original mode) and Softmax-specific computing (i.e., Softmax mode), increasing the flexibility of hardware resources.

[0045] In this embodiment, all sub-steps of the Softmax computation are merged into a small number of computation tasks, which are executed separately by the data processing unit and the tensor engine. A tree comparator array and repeat circuit are set in the data processing unit, a near-memory computation unit is set at the on-chip memory port, and a switchable data path multiplexer and a second write data path are set in the tensor engine. This reduces the Softmax computation, which requires 4 to 5 independent jobs in related technologies, to a small number of jobs, reducing task scheduling and synchronization overhead. Simultaneously, the near-memory computation unit performs subtraction operations along the path during the maximum value write-back to memory, eliminating the write-back and read-back operation of intermediate maximum values ​​and reducing memory bandwidth consumption. Furthermore, the multiplexer allows the tensor engine to flexibly switch between different modes, maximizing the reuse of existing hardware resources while accelerating Softmax and avoiding the area overhead caused by adding a large number of dedicated computation units for Softmax.

[0046] Based on any of the above embodiments, when the data processing unit processes the maximum value calculation and subtraction operation of the (i+1)th row of data, the tensor engine simultaneously processes the exponent lookup table and reduction summation operation of the i-th row of data; i is a non-negative integer, and i+1 is less than the total number of rows.

[0047] It should be noted that the pipeline timing of the data processing unit and the tensor engine is as follows: At time T0, the DAU begins processing the data of row 0. Since the difference data of row 0 has not yet been produced, the TENG is in an idle state; at time T1, the DAU has produced the difference data of row 0 and written it back to the OCM, and the DAU itself begins processing the data of row 1; at the same time, the TENG reads the difference data of row 0 from the OCM and begins to perform exponential lookup and reduction summation. At this time, the two pipelines begin to work in parallel; from T2 onwards, the DAU and TENG continue to process the data of adjacent rows in parallel. At each time step, the DAU is processing the preprocessing of the current row (row i+1), while the TENG is processing the postprocessing of the previous row (row i).

[0048] The two-stage pipeline parallelism solves the problem of idle execution units caused by data dependency in related technologies. In related technologies, after the DAU completes the maximum value calculation, it needs to wait for the TENG to complete the exponentiation calculation before it can proceed with subsequent processing. Even if the DAU has idle capacity, it cannot process the next row in advance. In this embodiment, however, the DAU and TENG process adjacent rows of data separately, without waiting for each other, thus making full use of hardware resources.

[0049] When the data volume M is large, the pipeline warm-up overhead (i.e., the idle time of TENG at time T0) can be ignored. The overall throughput is close to the slower processing speed of DAU and TENG, that is, the pipeline throughput is determined by the bottleneck level, which improves the throughput compared to non-pipelined solutions.

[0050] This embodiment limits i to a non-negative integer, and i+1 to less than the total number of rows, which effectively defines the effective range of pipeline parallelism: when i ranges from 0 to M At time 2, DAU processes the (i+1)th row while TENG processes the i-th row synchronously. That is, for the first M... For each pair of adjacent rows, the two pipelines remain in parallel; for the last row (i.e., i=M) 1) After DAU processing, TENG completes the finalization separately.

[0051] In this embodiment, by limiting the data processing unit to process the i+1th row of data while the tensor engine processes the i-th row of data, a two-stage pipeline parallel scheme is constructed, which enables the two execution units to overlap in execution on different rows of data, hides the waiting time caused by data dependency, and improves hardware utilization and computing throughput.

[0052] Based on any of the above embodiments, the tree comparator array is used to group the data elements of each row according to a preset number, and obtain the maximum value of each group through multiple levels of pairwise comparisons. Then, the maximum values ​​of each group are compared through a single level of comparison to obtain a scalar maximum value for each row.

[0053] It should be noted that, assuming the preset quantity is K, where K is a parameter determined during hardware design, the tree comparator array groups the K data elements in each row, and within each group, multiple levels of pairwise comparisons are performed to obtain the maximum value of each group. In some embodiments, K=16, dividing the N data elements in each row into N / 16 batches, and sending them batch by batch to the tree comparator array for processing. After each group of 16 data elements enters the tree comparator array, it undergoes pairwise comparisons by 8 tree comparators in the first stage to obtain 8 candidate values, 4 tree comparators in the second stage to obtain 4 candidate values, 2 tree comparators in the third stage to obtain 2 candidate values, and 1 tree comparator in the fourth stage to obtain the maximum value of the group. A total of 4 levels of comparison are required, with a delay of 4 clock cycles. After each group comparison is completed, the maximum value of each group is sent to the final comparator, and then undergoes another level of pairwise comparisons to compare the maximum values ​​of all groups, obtaining the final scalar maximum value for each row.

[0054] The group comparison design achieves an optimal balance between hardware area and computational speed. If all N elements were fed into the tree comparator array at once, it would require N... A single tree comparator has a huge hardware footprint and is impractical for large N values; if only two elements are compared each time, then N is needed. The single-stage serial comparison suffers from excessive latency. This embodiment employs a grouped comparison approach. Within each group, tree-based reduction achieves logarithmic latency, while between groups, multiplexing or cascading comparisons complete the final reduction, balancing hardware area and computational speed. For example, with N=1024 and K=16: each group of 16 elements requires 4 stages of comparison, totaling 64 groups; the comparison results from each group then undergo 6 stages of pairwise comparisons (64→32→16→8→4→2→1), requiring a total of 10 stages of comparison, far less than N. 1 = 1023 levels of serial comparison.

[0055] In this embodiment, a tree-based reduction structure is used to reduce the time complexity of finding the maximum value by grouping each row of data elements according to a preset number, obtaining the maximum value of each group through multi-level pairwise comparisons, and then obtaining the maximum value of the row through a single-level comparison. This achieves a balance between hardware area and computing speed.

[0056] Based on any of the above embodiments, the on-chip memory is an OCM on-chip memory, the near-memory computing unit is located at the data port of the OCM on-chip memory, and the subtractor is located near the OCM on-chip memory.

[0057] It should be noted that OCM (On-Chip Memory) is an SRAM memory integrated inside the NPU chip, used to store input data, intermediate results, and output data during the Softmax calculation process. Compared to off-chip DRAM, OCM has lower access latency (approximately 1-2 clock cycles) and higher bandwidth (synchronized with the processor core frequency).

[0058] The local memory computing unit is located at the data port of the OCM, that is, between the OCM's read / write data bus and the execution unit. The subtractor, as the computational component of the local memory computing unit, is physically located near the OCM, i.e., on the same chip. The distance between the subtractor and the OCM is controlled to ensure that the signal transmission delay is less than one clock cycle.

[0059] DAU will set the maximum value of each row to max(x). i The data is sent to the OCM via the write data path. Simultaneously, as the maximum value reaches the OCM write port, the near-memory computing unit reads the corresponding original input data x from the OCM read port. i The subtractor performs the subtraction operation x at the OCM port. i max(x i The difference data can then be written back to OCM via the OCM write port.

[0060] DAU only needs to write the maximum value to OCM and trigger the start of the near-memory computing unit. It does not need to participate in the subtraction operation itself. Therefore, DAU can immediately switch to the maximum value of the next row of data and perform the subtraction operation in parallel.

[0061] In this embodiment, by placing the near-memory computing unit at the data port of the OCM on-chip memory and the subtractor physically near the OCM, the entire process of reading the original data, subtracting the maximum value, and writing back the difference is completed near the memory port with full interface bandwidth. This does not occupy the core computing path of the data processing unit. It avoids the round-trip overhead of transporting the intermediate result of the maximum value over a long distance through the on-chip network to the remote execution unit and then writing it back. It also allows the data processing unit to start processing the next row of data immediately after writing the maximum value back to the OCM, and to execute it in parallel with the subtraction operation. This maximizes the release of memory bandwidth and eliminates the waiting time of the execution unit.

[0062] Based on any of the above embodiments, the multiplexer is disposed at the input end of the lookup table unit, the control end of the multiplexer is connected to the Job configuration register, the multiplexer includes a first input end, a second input end, and an output end, the first input end is connected to the data input port of the tensor engine, the second input end is connected to the output end of the reduction unit, and the output end of the multiplexer is connected to the input end of the lookup table unit; When the control bit is in the first state, the multiplexer selects the first input terminal to connect to the input terminal of the lookup unit, so that the input data is sent to the lookup unit for exponential lookup, the lookup result is output through the first write data path, and the output of the lookup unit is sent to the reduction unit for summation and then output through the second write data path. When the control bit is in the second state, the multiplexer selects the second input terminal to connect to the input terminal of the lookup unit, so that the input data is processed by the reduction unit and then sent to the lookup unit for lookup, and the lookup result is output through the first write data path.

[0063] It should be noted that a multiplexer is a selection circuit with multiple data inputs, one data output, and one control input. Its function is to connect one of the multiple data inputs to the data output based on the selection signal received from the control input.

[0064] In this embodiment, the hardware connection relationship of the multiplexer is as follows: The first input terminal is connected to the data input port of the tensor engine. This data input port receives the difference data read from the on-chip memory (OCM), that is, the x output by the DAU after completing the maximum value calculation and near-memory subtraction. i max(x i ).

[0065] The second input terminal is connected to the output terminal of the reduction unit. This input receives the data processed by the reduction unit. In the original mode (i.e., when the control bit in the Job configuration register is in the second state), the reduction unit reduces the input data, such as performing operations like summation or finding the maximum value. The result after reduction is sent to the second input terminal of the multiplexer through the output terminal of the reduction unit.

[0066] The fact that the output of the multiplexer is connected to the input of the lookup unit means that regardless of which input the multiplexer selects, its output will ultimately be sent to the lookup unit for the lookup operation. The function of the multiplexer is to determine the source of the input data for the lookup unit: whether it comes from the raw difference data from the tensor engine's data input port or from the data processed by the reduction unit.

[0067] The multiplexer's control terminal is connected to the Job configuration register. The Job configuration register is a group of control registers in the NPU used to store the current Job configuration information. Before starting a Job, the scheduler writes parameters such as the Job type, data address, and data shape into the Job configuration register. In some embodiments, a control bit is added to the Job configuration register specifically for controlling the multiplexer's data path switching. The physical output of this control bit is connected to the multiplexer's control terminal.

[0068] First state: Softmax mode. When the control position in the Job configuration register is in the first state (e.g., logic high level 1), the multiplexer selects the first input terminal to connect to its output terminal, and then to the input terminal of the lookup table unit. In this state, the tensor engine executes in Softmax mode, and the specific data flow is as follows: Step 1: Data Input. The Tensor Engine's data input port receives the difference data x from the on-chip memory. i max(x i The data enters the multiplexer through its first input and is then selected to connect to the lookup table unit.

[0069] Step 2: Exponential Lookup. The difference data is entered into a lookup table (LUT). The lookup table internally stores an approximate value table of the exponential function. Using the difference data as the index address, the lookup is performed, and the corresponding exponential value exp(x) is output. i max(x i )).

[0070] Step 3: Dual-path parallel output. The output of the lookup unit, i.e., the exponent value, is branched into two paths within TENG: First path: written to on-chip memory via the first write data path (WDMA0). This path outputs the exponent value corresponding to each data element in each row, with a shape of (M, N). Second path: fed into the reduction unit for summation. The reduction unit performs element-wise summation on the N exponent values ​​in the same row to obtain the exponent sum for that row. After summation, the exponent sum is written to on-chip memory via the second write data path (WDMA1). This path outputs a scalar value for each row, with a shape of (M, 1).

[0071] In Softmax mode, the first and second write data paths operate simultaneously, outputting the exponent value and the exponent sum, respectively. Since the first write data path is the original write data path of TENG, and the second write data path is a newly added write data path, the two share the physical bandwidth through time-division multiplexing to ensure that the two outputs do not interfere with each other.

[0072] Second State: Original Mode. When the control position in the Job configuration register is in the second state (e.g., logic low 0), the multiplexer selects the second input to connect to its output, and then to the input of the lookup table unit. In this state, the Tensor Engine executes in original mode, restoring the original data path sequence of TENG. The specific data flow is as follows: Step 1: Data Input and Reduction. The Tensor Engine's data input port receives input data. This data is first sent to the reduction unit for processing. The reduction unit performs the corresponding reduction operation (such as summation, finding the maximum value, finding the average value, etc.) according to the Job configuration, and outputs the result to the second input of the multiplexer after reduction.

[0073] Step 2: Data Selection and Table Lookup. Under the control of the control bit, the multiplexer selects the second input terminal to connect to its output terminal, and sends the output data of the reduction unit to the table lookup unit for table lookup operation.

[0074] Step 3: Output the lookup results. After the lookup is completed, the lookup results are written to the on-chip memory via the first write data path (WDMA0).

[0075] In raw mode, the second write data path (WDMA1) is not active or is engaged in other non-Softmax tasks. The TENG's reduction unit and adder share the same hardware resources, so the reduction operation performed in raw mode shares hardware resources with the summation operation in Softmax mode.

[0076] In this embodiment, by setting the multiplexer at the input of the lookup unit and using a single control bit in the Job configuration register to switch between the two data paths, the tensor engine achieves the fusion of exponential lookup and reduction summation in a single job by simply adding a multiplexer and a write data path. At the same time, it retains TENG's full support for general tensor operations, achieving both Softmax-specific and general computing capabilities with minimal hardware area increment.

[0077] Based on any of the above embodiments, the first calculation task includes: the data processing unit calculates the maximum value of each row of data, copies it to the same shape as the input data through the repeat circuit, and writes it back to the on-chip memory; at the same time, the near-memory calculation unit is triggered to perform a subtraction operation to obtain the difference data and write it back to the on-chip memory; the difference data is used as the output of the first calculation task. The second computation task includes: the tensor engine reads the difference data from the on-chip memory, performs exponent lookup and reduction summation operations in the Softmax mode, and outputs the exponent value and the exponent sum.

[0078] Specifically, the first computation task is performed by the data processing unit and includes the following steps: The first step is that the data processing unit receives input data X of shape (M, N). For the i-th row (i = 0, 1, ..., M) 1) A tree comparator array compares the N elements of the row in parallel to obtain the scalar maximum value max(x). i ).

[0079] The second step is to use the repeat circuit to max(x) i The vector is copied N times to form a maximum value vector of shape (1, N). After copying, this maximum value vector is sent to the on-chip memory through the write data path.

[0080] Thirdly, during the process of writing the maximum value vector back to on-chip memory, the near-memory computation unit located at the on-chip memory port is triggered. The near-memory computation unit reads back the original input data x for that row from the on-chip memory. i The subtraction operation x is performed at the memory port. i max(x i ), thus obtaining the difference data.

[0081] The fourth step is to write the difference data back to the on-chip memory through the write data path of the on-chip memory.

[0082] Fifth, the difference data, as the final output of the first computing task, is stored in the on-chip memory for the second computing task to read.

[0083] The second computational task is performed by the tensor engine and includes the following steps: The first step is for the tensor engine to read the difference data x from the on-chip memory. i max(x i ).

[0084] The second step, in Softmax mode, involves the difference data being entered into a lookup table for exponent lookup, yielding the exponent value exp(x). i max(x i )).

[0085] The third step is to output the exponent value to the on-chip memory through the first write data path.

[0086] Fourthly, the exponent value is also fed into the reduction unit for element-wise summation to obtain the exponent sum of the row.

[0087] Fifth, the exponent and the output are sent to the on-chip memory via the second write data path.

[0088] The first computation task outputs the difference data, and the second computation task reads the difference data as input. There is a data dependency between the two computation tasks; the second computation task must wait for the first computation task to complete the output of the difference data for the corresponding row before it can begin processing. However, this data dependency does not cause idle waiting by the execution units; that is, while the DAU is processing the (i+1)th row, the TENG is simultaneously processing the ith row. The data dependency is hidden within the overlapping parallelism of the execution units.

[0089] In this embodiment, the boundaries of the sub-steps completed by each of the two computing tasks and the direction of the output data flow are defined, enabling the scheduler to complete the issuance of the two jobs with minimal information configuration. At the same time, it ensures that there is a deterministic data flow connection between the storage location of the difference data output by the first computing task in the on-chip memory and the read address of the second computing task, providing timing guarantees for pipeline parallelism.

[0090] Based on any of the above embodiments, the hardware accelerator further includes: The reciprocal and normalization execution unit is used to calculate the reciprocal of the exponent and the sum, and multiply the exponent value by the reciprocal to obtain the Softmax normalization result.

[0091] It should be noted that the reciprocal and normalization execution units can be independent hardware modules or reuse the internal computing resources of TENG.

[0092] The input to the reciprocal and normalization execution unit is the exponent sum (scalar value) of the output of the second computation task, and the output is the Softmax normalized result.

[0093] The reciprocal operation is the reciprocal of the exponent sum. Since the exponent sum is a scalar, one per line, the computational cost of finding the reciprocal is very small.

[0094] In some embodiments, the reciprocal operation is performed by looking up a reciprocal table. The reciprocal and normalization execution unit internally stores a table of approximate values ​​of the reciprocal function, and the corresponding reciprocal is obtained by looking up the table using the exponent and sum as index addresses. Compared to direct division, looking up the reciprocal table has lower latency and requires less hardware area.

[0095] In some embodiments, the reciprocal operation is performed using a lightweight divider. The lightweight divider is a fixed-point divider that requires only a few clock cycles to complete a division operation.

[0096] The normalization operation is to multiply each exponent value by its reciprocal.

[0097] The reciprocal and normalization execution unit contains a multiplier array that can perform multiple multiplication operations in parallel. For N exponent values ​​in a row, the multiplier array multiplies each exponent value by its reciprocal in parallel, resulting in N normalized results.

[0098] In some embodiments, the reciprocal operation can be performed directly within the Tensor Engine using a lookup unit. The TENG lookup unit stores both the exponent table and the reciprocal table; when the input is an exponent value, the exponent table is looked up, and when the input is a sum of exponents, the reciprocal table is looked up. This eliminates the need for separate reciprocal and normalization execution units.

[0099] The reciprocal and normalization operations begin after the second computation task completes the exponent and output. Since each row of the reciprocal and normalization operations only requires one division (or table lookup) and N multiplications, the computational load is much smaller than that of exponent table lookup and reduction summation. Therefore, this step will not become the performance bottleneck of the entire Softmax computation.

[0100] In this embodiment, by setting up independent execution units for reciprocal calculation and normalization, the two sub-steps of reciprocal calculation and normalization multiplication in Softmax calculation are separated from the tensor engine and completed by dedicated hardware. This allows the tensor engine to start processing the next batch of data after completing the exponent lookup table and reduction summation output, without waiting for the reciprocal calculation and normalization operations to complete.

[0101] Based on any of the above embodiments, the shape of the input data is (M, N); where M is the batch processing dimension and N is the sequence length dimension, and the N data elements in each row are divided into multiple batches and input into the tree comparator array.

[0102] It's important to note that in the Transformer's attention mechanism, Softmax is typically applied to each row of the attention score matrix. That is, for an input matrix of shape (M, N), Softmax normalization is performed on each of the M rows. M corresponds to the batch size, and N corresponds to the sequence length. For example, in a typical Transformer model, M=1 (single-sample inference) or M=4 / 8 (batch inference), and N=512 / 1024 / 2048 (sequence length).

[0103] The N data elements in each row are divided into multiple batches and fed into a tree-structured comparator array. The reason for batching is that the parallelism of the hardware comparator array is limited by the chip area; comparing all N elements simultaneously would require N... A single comparator is impractical when N is large.

[0104] In some embodiments, K data elements are fed in each batch (e.g., K=16), for a total of N / K batches. The first batch of K elements is fed into a tree comparator array for intra-group reduction to obtain the maximum value of the batch; the second batch of K elements is fed into the tree comparator array to obtain the maximum value of the second batch; and so on. After all batches are processed, the maximum values ​​of each batch are compared again to obtain the final maximum value of the entire row.

[0105] Batch processing trades time for area, performing reduction operations on large vectors with a smaller hardware footprint. For N=1024 and K=16, only 16 comparators (forming a tree-like reduction structure) are needed to complete all comparisons, with the hardware area being only 1 / 64 of the fully parallel solution. Meanwhile, inter-batch comparisons (comparisons between K maximum values) require only an additional stage of comparison, resulting in a minimal increase in latency.

[0106] In this embodiment, each batch of data enters the comparator array sequentially for intra-group reduction, and the maximum value of each group is then compared at the final stage to obtain the row maximum value. When N is much larger than the preset number of groups, this batch processing method enables the limited comparator hardware resources to process sequences of arbitrary lengths, adapts to different sequence length configurations in Transformer, and has good scalability.

[0107] Furthermore, based on the same concept as the aforementioned hardware accelerator based on Softmax computation, this application embodiment also provides a Softmax hardware acceleration method based on the combination of execution unit pipeline (EU pipeline) and near-memory computation.

[0108] In this embodiment, the Softmax hardware acceleration method aims to solve a series of problems caused by the excessive number of jobs when general NPU hardware performs Softmax operations, such as: Problem 1: High scheduling overhead. The startup, scheduling, and synchronization of each job require additional control overhead. In models such as Transformer, Softmax is called frequently, and the excessive number of jobs causes the scheduler to intervene frequently, seriously affecting hardware execution efficiency. Problem 2: Repeated handling of intermediate results leads to memory access bottlenecks. In related technologies, there are strict data dependencies between jobs, and the output of each job must be written back to memory, and then read again by the next job. For example, the result of finding the maximum value needs to be written back to memory and then read by the exponential offset job, and the result of the exponential offset needs to be written back to memory and then read by the exponential sum job. This repeated process of calculation-write-read consumes a lot of memory bandwidth, forming a memory access bottleneck. Problem 3: Low utilization of execution units. Due to the data dependencies between jobs, adjacent steps cannot be merged, and each execution unit is idle while waiting for the previous job to complete, and the hardware parallel capability is not fully utilized.

[0109] The Softmax hardware acceleration method includes: Using near-memory calculations to combine maximum value and subtraction operations: For problems one and two, this embodiment combines the maximum value calculation and subtraction operation into a single job in DAU. The specific method is as follows: See Figure 1 , Figure 1 This is a schematic diagram of a DAU hardware microarchitecture provided in an embodiment of this application. A tree-structured comparator array is added inside the DAU to perform parallel comparisons on each row of the input data and obtain the maximum value for each row. Then, the data is written back to the OCM (On-Chip Memory). Subsequently, using the near-memory computing unit near the OCM memory, the original data is read from the OCM at full memory bandwidth. → Subtract → Write the difference result back to the OCM subtraction operation.

[0110] The key to near-memory computation lies in the fact that subtraction operations are performed near the memory port of the OCM, and both reads and writes are performed at full efficiency of the memory interface, without occupying the core computation path of the DAU. Therefore, finding the maximum value and subtraction operations can be efficiently completed within the same job, without having to write the maximum value back to memory as the output of an independent job and then read it by the next job, thus merging two originally independent jobs into one.

[0111] Implementing parallel operation of DAU and TENG using EU pipeline: To address issue three, this embodiment utilizes two execution units, DAU and TENG, to construct a two-stage pipeline: DAU handles the initial computation, including finding the maximum value and performing near-memory subtraction preprocessing; TENG handles the subsequent computation, including exponent lookup and reduction summation. When DAU processes the maximum value and subtraction of the (i+1)th row of data, TENG can simultaneously process the exponentiation and summation operations of the ith row. The two execution units alternately process data from different rows, forming a stable pipeline that effectively hides the waiting time caused by data dependencies and significantly improves the utilization rate of the execution units.

[0112] Modify TENG data path merging index lookup and reduction summation: To address question four, this embodiment modifies the internal data path of TENG, merging the exponent lookup table and reduction summation into a single job within TENG. Specifically, a multiplexer (MUX) is added within TENG to support switching between the original computation order (input → multiplication → reduce → LUT → WDMA0) and the Softmax computation order (input → LUT → WDMA0, LUT output → reduce → WDMA1). A second write data path (WDMA1) is also added for independently outputting the reduction summation result.

[0113] In Softmax mode, TENG first performs a LUT lookup on the input data to complete the exponentiation operation. The lookup result is written to memory via WDMA0 and simultaneously sent to the reduction unit for summation. The summation result is written out using WDMA1. Thus, TENG completes both the exponent calculation and the reduction summation steps within a single job.

[0114] Overall Job Process: After the above optimizations, the Softmax calculation process in this embodiment is simplified to two jobs: New Job0: DAU Execution The data is retrieved and written back to the OCM, and then the near-memory computing units near the OCM memory are used to complete the computation at full bandwidth. The subtraction operation results in a value that is written back to the OCM and then output by the DAU in the form of (M, N).

[0115] New Job 1: TENG reads the difference data from OCM and completes the index lookup in sequence. Summation with specifications The final normalized result is output within a single Job.

[0116] DAU and TENG form a two-stage pipeline that alternates and runs in parallel when processing multiple rows of data, further improving throughput.

[0117] This embodiment also provides a neural network processing unit (NPU) architecture that includes multiple execution units, comprising the following hardware modules: DAU (Data Arithmetic Unit): Responsible for data preprocessing operations, including comparisons and simple arithmetic operations. In this embodiment, the DAU is equipped with circuits for finding the maximum value and data repeating, and works with the near-memory computing unit near the OCM memory to complete subtraction operations.

[0118] TENG (Tensor Engine): Responsible for complex operations at the tensor level, including multiplication, lookup table (LUT) operations, and reduction operations. In this embodiment, TENG adds data path multiplexing capabilities.

[0119] OCM (On-Chip Memory): Responsible for data storage. DAU and TENG (Execution Unit) and other EUs (Execution Units) read input data from memory or write calculation results from the execution unit to memory through their respective RDMA (Read DMA) and WDMA (Write DMA).

[0120] A set of parallel comparator circuits is added inside the DAU to perform element-by-element comparisons on the input data vector to find the maximum value. The specific implementation is as follows: The DAU receives input data of shape (M, N), where M is the batch size and N is the sequence length. For each row (M rows in total), the DAU divides the 16 data elements into N / 16 batches and feeds them into the comparator array. The comparator array uses a tree-like reduction structure: the first level compares each of the 16 elements pairwise to obtain 8 candidate maximum values; the second level compares each of the 8 candidate values ​​pairwise to obtain 4 candidate values; and so on recursively. After the first-level comparison, the maximum value of the 16 numbers is obtained. Then, a second-level inter-step comparison is performed to compare the N / 16 values ​​in each row to find the maximum value, and finally obtain the maximum value with shape (M, 1).

[0121] A repeat module is added inside the DAU, following the maximum value module. This module copies the maximum value of each row N times, thus repeating the maximum value of shape(M,1) into shape(M,N), allowing it to be integrated with the original tensor. Subtraction.

[0122] DAU calculates the maximum value for each row. After repeating the operation, the maximum value is written back to the OCM. During this process, the subtraction operation is performed along the path using the nearby memory units in the OCM. The specific steps are as follows: Step 1: The near-memory computing unit receives the data written by the DAU. Simultaneously, the original input data for that line is read from the OCM at full bandwidth. ; Step 2: The near-memory computing unit performs a subtraction operation near the OCM memory port. The difference is obtained. The subtractor is located near the memory of the OCM, and read and write operations are performed at full efficiency of the memory interface, without occupying the DAU core computing path; Step 3: Write the difference back to the OCM at full bandwidth.

[0123] The advantage of this near-memory computing scheme is that the subtraction operation is completed at full efficiency near the OCM memory. The entire process of reading the original data → subtracting the maximum value → writing back the difference is bottlenecked by the memory interface bandwidth and does not consume additional computation cycles of the execution unit. Therefore, the subtraction process can be completed efficiently in the same job where the maximum value is obtained in the DAU.

[0124] The original data path of TENG is: input data → reduction unit (Reduce / Add) → lookup unit (LUT) → write data path (WDMA0). The reduction unit (Reduce) and adder (Add) are multiplexed hardware resources, and only one operation, reduction or addition, can be performed at a time. This means that in the original architecture, TENG cannot simultaneously complete the lookup-then-reduce operation sequence (i.e., LUT followed by Reduce) within a single job because the fixed order of the data path limits the flexibility of computation. For more information, see... Figure 2 , Figure 2 This is a schematic diagram of a TENG hardware microarchitecture provided for an embodiment of this application. The data path of the TENG is modified in this embodiment as follows: A second write data path (WDMA1) is added: An independent write data path, WDMA1, is added within TENG to write out the intermediate results after the reduction summation is completed. WDMA1 shares the physical bandwidth with the existing WDMA0 and is used alternately through time-division multiplexing.

[0125] Added Data Path Sequence Switching (MUX): A multiplexer (MUX) has been added inside TENG to support switching between two computation sequences: Sequence A (representing the original mode): Input → Reduce → LUT → WDMA0. Sequence B (representing the Softmax mode): Input → LUT → WDMA0, while LUT output → Reduce → WDMA1. In Sequence B, TENG first performs a LUT lookup (exponential operation) on the input data. The lookup result is written to memory via WDMA0 as the output of the exponent value; on the other hand, the lookup result is sent to the reduction unit for summation, and the summation result is written out via WDMA1.

[0126] The switching of the MUX is determined by a control bit in the Job configuration register, namely Figure 2 The sel signal in the context of Softmax computing. In the Softmax computing scenario, when the scheduler configures the TENG Job, it sets this control bit to 1, and the TENG executes in order B; in other scenarios, the control bit is 0, and the TENG executes in the original order A.

[0127] The complete Softmax calculation process is described using an input data shape of (M, N) as an example: Job0: DAU performs maximum value calculation and near-term subtraction; Input: Original input data X, shape (M, N), data format is BF16.

[0128] Execution process: (a) DAU will assign each row i (i = 0, 1, ..., M) to a specific value. 1) The N elements are fed into a tree comparator array to obtain the maximum value of the row. (b) Maximum value After repeating, the shape is restored to (M, N), and then written back to OCM via WDMA. (c) Using the nearest memory unit near the OCM memory, the original data of that row is read from the OCM. Perform subtraction And write the difference back to OCM.

[0129] Output: Difference data Shape (M, N), data format is BF16.

[0130] Job 1: TENG execution index lookup and reduction summation: Input: Difference data The data has a shape (M, N) and a data format of BF16. This data has been written back to the OCM by the DAU through near-memory computation, and the TENG reads it from the OCM.

[0131] Execution process: TENG executes in sequential B mode, for each line i (i = 0, 1, ..., M) 1) Perform the following operations: 1. Exponential Lookup Table (LUT): Input the N difference data of this row into the LUT lookup table unit. The LUT table stores the approximate value of the exponential function. Input the difference in BF16 format and output the corresponding exponential value. The lookup result is sent to two paths simultaneously: one path writes it to the OCM via WDMA0, and the other path sends it to the reduction unit. 2. Reduction and Summation: The reduction unit receives the exponent values ​​output by the LUT and calculates the exponent sum for each row by adding them element by element. The summation result after reduction is written to OCM via WDMA1. 3. Calculating the reciprocal and final normalization: Since the computational cost of calculating the reciprocal is very small (only one scalar per line), the reciprocal operation can be performed internally within TENG by looking up the reciprocal table using a LUT, or by a separate lightweight divider. Finally, TENG or a subsequent lightweight job initiated by the scheduler multiplies the exponent by the reciprocal to obtain the final normalized result.

[0132] Output: Softmax normalized result, shape (M, N), data format BF16.

[0133] The pipeline timing between DAU and TENG is shown in Table 1 below: Table 1

[0134] At time T0, the DAU begins processing the 0th row of data, calculating the maximum value and performing subtraction preprocessing. During this time, the TENG is idle, waiting for the DAU to output the difference data from the 0th row. At time T1, the DAU processes the 1st row of data, while the TENG receives the difference data from the 0th row and begins exponential lookup and reduction summation. In each subsequent time step, the DAU and TENG alternately process adjacent rows of data, forming a stable two-stage pipeline. When the data volume is large (M is large), the pipeline warm-up overhead, i.e., the TENG's idle time at T0, is negligible, and the overall throughput is close to the processing speed of the slower of the DAU or TENG.

[0135] In this embodiment, the number of jobs is significantly reduced, with Softmax computation reduced from 4-5 jobs to 2 jobs. The startup, scheduling, and synchronization overhead of each job is reduced accordingly, significantly lowering the scheduler's burden. Data transfer overhead is significantly reduced. By using near-memory computation to find the intermediate result between the maximum value and subtraction operation, the maximum value does not need to pass through memory; the subtraction operation is performed directly near the OCM memory with full bandwidth, eliminating memory write-back and read operations for intermediate results and saving memory bandwidth. Execution unit utilization is improved. Through the two-stage pipeline design of DAU and TENG, the two execution units work in parallel when processing different rows of data, effectively eliminating the idle waiting time of execution units caused by data dependencies, and fully utilizing the hardware parallelism capability. The inference performance of Transformer models is significantly improved. Softmax is frequently called in the Transformer's self-attention mechanism; this embodiment greatly reduces its latency and improves the end-to-end inference performance of large language models, visual Transformers, and other models.

[0136] Based on the hardware accelerator described in any of the above embodiments, this application also provides a neural network processor, including the hardware accelerator described in any of the above embodiments.

[0137] The Neural Processing Unit (NPU) is a dedicated processor chip for neural network inference computation. Its functions include accelerating matrix multiplication, convolution, activation function, and other operations in neural network models. The NPU includes the hardware accelerator described in any of the above embodiments as one of its dedicated computing units. When the processor executes inference for a Transformer model, the Softmax computation task is scheduled to be executed by the hardware accelerator, while other computation tasks (such as matrix multiplication, LayerNorm, etc.) are executed by other computing units within the processor. The NPU also includes modules such as a task scheduler, on-chip memory (OCM), a data bus, and an interrupt controller. The task scheduler determines the location of the Softmax operator based on the currently executed model type (Transformer, BERT, GPT, etc.) and distributes the Softmax computation task to the DAU and TENG as a first computation task and a second computation task, respectively.

[0138] Based on the hardware accelerator described in any of the above embodiments, this application also provides a chip, on which the hardware accelerator described in any of the above embodiments or the neural network processor described in any of the above embodiments is integrated.

[0139] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0140] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0141] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0142] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0144] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A hardware accelerator based on Softmax operation, characterized in that, The hardware accelerator includes: The execution unit pipeline includes a data processing unit and a tensor engine. The data processing unit is used to execute a first computation task, and the tensor engine is used to execute a second computation task. The first computation task and the second computation task together cover the maximum value operation sub-step, the exponentiation operation sub-step, the summation operation sub-step, the reciprocal operation sub-step, and the normalization operation sub-step. The data processing unit includes a tree comparator array and a repeat circuit. The tree comparator array is used to perform parallel comparisons on each row of the input data to obtain the maximum value of each row. The repeat circuit is used to copy the maximum value of each row to the same shape as the input data and write it back to the on-chip memory. A proximity calculation unit is provided at the port of the on-chip memory. The proximity calculation unit is used to read the input data from the on-chip memory at full bandwidth of the memory interface and perform a subtraction operation to obtain the difference data during the process of writing the maximum value of each row back to the on-chip memory, and write the difference data back to the on-chip memory. The tensor engine includes a lookup unit, a reduction unit, a first write data path, a second write data path, and a multiplexer. The multiplexer is used to switch the data path in Softmax mode as follows: the input data is sequentially processed by the lookup unit for exponent lookup, one path outputs the exponent value through the first write data path, and the other path is processed by the reduction unit for summation and then outputs the exponent sum through the second write data path.

2. The hardware accelerator according to claim 1, characterized in that, When the data processing unit processes the maximum value and subtraction operation of the (i+1)th row of data, the tensor engine simultaneously processes the exponent lookup table and reduction summation operation of the i-th row of data; i is a non-negative integer, and i+1 is less than the total number of rows.

3. The hardware accelerator according to claim 1, characterized in that, The tree comparator array is used to group the data elements of each row according to a preset number. Within each group, the maximum value of each group is obtained through multiple levels of pairwise comparisons. Then, the maximum values ​​of each group are compared through a single level of comparison to obtain a scalar maximum value for each row.

4. The hardware accelerator according to claim 1, characterized in that, The on-chip memory is an OCM on-chip memory, the near-memory computing unit is located at the data port of the OCM on-chip memory, and the subtractor is located near the OCM on-chip memory.

5. The hardware accelerator according to claim 1, characterized in that, The multiplexer is located at the input of the lookup table unit. The control terminal of the multiplexer is connected to the Job configuration register. The multiplexer includes a first input, a second input, and an output. The first input is connected to the data input port of the tensor engine, the second input is connected to the output of the reduction unit, and the output of the multiplexer is connected to the input of the lookup table unit. When the control bit is in the first state, the multiplexer selects the first input terminal to connect to the input terminal of the lookup unit, so that the input data is sent to the lookup unit for exponential lookup, the lookup result is output through the first write data path, and the output of the lookup unit is sent to the reduction unit for summation and then output through the second write data path. When the control bit is in the second state, the multiplexer selects the second input terminal to connect to the input terminal of the lookup unit, so that the input data is processed by the reduction unit and then sent to the lookup unit for lookup, and the lookup result is output through the first write data path.

6. The hardware accelerator according to claim 1, characterized in that, The first calculation task includes: the data processing unit calculates the maximum value of each row of data, copies it to the same shape as the input data through the repeat circuit, and writes it back to the on-chip memory; at the same time, the near-memory calculation unit is triggered to perform a subtraction operation to obtain the difference data and write it back to the on-chip memory; the difference data is used as the output of the first calculation task. The second computation task includes: the tensor engine reads the difference data from the on-chip memory, performs exponent lookup and reduction summation operations in the Softmax mode, and outputs the exponent value and the exponent sum.

7. The hardware accelerator according to claim 1, characterized in that, The hardware accelerator also includes: The reciprocal and normalization execution unit is used to calculate the reciprocal of the exponent and the sum, and multiply the exponent value by the reciprocal to obtain the Softmax normalization result.

8. The hardware accelerator according to claim 1, characterized in that, The shape of the input data is (M, N); where M is the batch processing dimension and N is the sequence length dimension, and the N data elements in each row are divided into multiple batches and input into the tree comparator array.

9. A neural network processor, characterized in that, Includes the hardware accelerator according to any one of claims 1-8.

10. A chip, characterized in that, The chip integrates the hardware accelerator of any one of claims 1-8, or the neural network processor of claim 9.

Images