System employing converter architecture in combination with quantitative perceptual non-linear approximation and near storage computation

Abstract

The invention provides a GQA-LUT method. According to the GQA-LUT method, a non-linear operator in a converter is effectively approximated by utilizing a genetic algorithm and an LUT-based circuit. The GQA-LUT method can adaptively find optimal solutions of various nonlinear functions, and is superior to a conventional neural network method. A novel rounding variation (RM) algorithm enhances approximation accuracy during quantization, thereby improving low order integer precision. In the invention, a LayerNorm folding strategy is also introduced as a near memory calculation principle, so that IO and energy overhead of the hierarchical structure of the two-stage memory is reduced. Furthermore, an additive partial and quantization method is proposed to reduce energy consumption by quantizing accumulated PSUM in matrix multiplication as well as a PSQ-APSQ grouping policy and floating point regularization.

Description

Technical Field

[0001] The present invention relates to efficient hardware implementation for nonlinear operations in converter models. More specifically, the present invention relates to a genetic LUT approximation algorithm (GQA-LUT) and a near-memory architecture for INT8 quantization. Background Art

[0002] The introduction of transformer-based neural networks has revolutionized the fields of natural language processing and computer vision, primarily due to the powerful self-attention mechanism that efficiently captures long-range dependencies. However, this technological breakthrough comes at the cost of increased computational power and memory. In response, current research has focused extensively on making transformers more suitable for deployment on edge devices. Furthermore, an integrated approach has emerged, combining convolution with linear attention as a lightweight architecture strategy. Furthermore, techniques such as quantization and runtime pruning are gaining increasing attention to reduce hardware requirements.

[0003] Despite these efforts, optimizing nonlinear operations within transformers remains an underdeveloped area and often relies on high-precision arithmetic capabilities (e.g., 32-bit floating-point (FP32) or integer (INT32) operations), significantly increasing hardware costs. According to several work reports, the low efficiency of nonlinear operations has severely hindered the development of transformer accelerators on many hardware platforms. Furthermore, in transformer-based accelerators, operations such as Softmax and LayerNorm cause large amounts of data movement between on-chip SRAM and the computational core, adversely affecting power efficiency and processing speed.

[0004] In the field of specialized neural network accelerators, quantization schemes using single-precision (e.g., INT8) or mixed bit formats have become very common. These compression methods typically use a binary arithmetic pipeline for integer-only calculations and several scaling factors.

[0005] Many studies have been devoted to addressing the computational challenges associated with nonlinear operations in neural networks. For example, related work has introduced a method called I-BERT to approximate GELU, Softmax, and LayerNorm functions using quantization-aware 32-bit integer (INT32) arithmetic.

[0006] To further reduce the hardware resource consumption and power consumption of Softmax, related work has also designed a method based on long sum exponentials and used piecewise linear approximation (PLA) to repeatedly approximate the exponential and logarithmic functions. In addition, related work has also developed a low-precision Softmax technique using a basis replacement strategy.

[0007] In addition, related work has proposed deep neural network architectures using PLA based on floating-point precision. Although these methods provide a way to accelerate nonlinear operations with minimal impact on accuracy, they lack versatility because each optimized operator requires a unique computational data flow.

[0008] To this end, a general-purpose PLA framework based on a neural network, NN-LUT, has been proposed in the related field to approximate any nonlinear function and store the parameters in a lookup table (LUT). However, this framework still relies on a large number of parameters that are independent of the input data range, resulting in inefficient hardware utilization because the optimal parameters are different for various input data ranges. Research on optimizing these methods includes simplifying the piecewise linear approximation (PLA) based on the lookup table (LUT) by factoring the floating-point data and applying a single-entry approximation calculation to the mantissa within a limited range.

[0009] However, when high-precision approximation techniques are applied to INT8 inputs, they lead to inefficient use of resources due to the significantly lower expressiveness of INT8 compared to FP / INT32. Given the above, to date, no method has been developed that can effectively utilize low-order integer (INT8) precision while understanding the effects of quantization.

[0010] Therefore, there is a need for a method that can fully understand the quantization effect, utilize low-bit integer (such as INT8) precision, and effectively optimize the nonlinear operations in the converter model, thereby improving hardware efficiency and reducing resource consumption. Summary of the Invention

[0011] An object of the present invention is to provide a method and system that address the aforementioned shortcomings and unmet needs in the state of the art.

[0012] According to a first aspect of the present invention, a system is provided that employs an improved approximation architecture and utilizes a piecewise linear approximation method to implement an efficient transformer acceleration program. The system includes a LUT-based approximation circuit, a quantization-aware enhancement module, a low-bit LUT-based piecewise linear approximation circuit, and a memory module. The LUT-based approximation circuit is configured to adaptively approximate a nonlinear operator using a genetic algorithm. The genetic algorithm evaluates multiple candidate solutions to generate optimal approximation parameters. The LUT-based approximation circuit converts at least one original floating-point operation into at least one fixed-point operation using parameters, such that the LUT-based approximation circuit outputs a transformed fixed-point representation of the original floating-point operation based on the approximation parameters. The quantization-aware enhancement module is configured to execute a rounding mutation algorithm that treats the conversion from floating-point to integer as a mutation process. The quantization-aware enhancement module works in conjunction with the LUT-based approximation circuit to round the fixed-point output generated by the LUT-based approximation circuit to minimize quantization error while maintaining accuracy, thereby causing the quantization-aware enhancement module to output a quantized integer representation of the transformed fixed-point operation. The low-order LUT-based piecewise linear approximation circuit is configured to store all approximation parameters in INT8 format, thereby reducing memory and computational overhead. The low-order LUT-based piecewise linear approximation circuit receives the quantized integer representation from the quantization-aware enhancement module and performs piecewise linear approximation, so that the low-order LUT-based piecewise linear approximation circuit outputs a low-order approximation of the nonlinear operation. The storage module is configured to store the low-order approximation value of the nonlinear operation in the efficient storage INT8 format.

[0013] According to a second aspect of the present invention, a near-memory computation engine for Softmax and LayerNorm operations is provided. The near-memory computation engine comprises a near-memory architecture model, a piecewise linear approximation engine, and a LayerNorm folding module. The near-memory architecture model is customized to reduce the data movement burden associated with Softmax and LayerNorm computations. The piecewise linear approximation engine is implemented via a LUT-based approximation circuit and is configured to utilize a lookup table and quantization method to minimize data transfer and enhance overall computational efficiency. The LayerNorm folding module is used to reduce the required parameters and computation steps.

[0014] Through the above configuration, the present invention provides at least three novel features, which are summarized as follows:

[0015] (1): A LUT approximation technique called GQA-LUT that uses a genetic algorithm to perform approximation on any nonlinear operation.

[0016] (2) A quantization-aware rounding mutation (RM) algorithm is proposed to improve the accuracy of the approximation by treating quantization as a specific mutation process in the genetic algorithm.

[0017] (3) To alleviate the data movement burden caused by Softmax and LayerNorm, a near-memory computing-based PLA engine is proposed to reduce the huge data movement between the computation module and the global buffer (GBuf). In addition, a LayerNorm folding strategy is proposed to convert element-by-element transformations into convolutions, thereby reducing the energy consumption of buffer access and computation.

[0018] In addition, a new additive partial sum (PSUM) quantization strategy, called APSQ, is introduced to specifically address the challenges associated with large PSUMs in weight static (WS) and input static (IS). In APSQ, each quantizer can be enabled to consider the accumulated prior results instead of focusing only on the current PSUM. In addition, a grouping strategy is introduced to combine APSQ and PSQ with floating-point alignment regularization to avoid redundant rounding of PSUM due to its additive nature. The strategy includes (i) a new additive partial sum quantization method (APSQ), which incorporates the accumulation effect into the quantizer; and (ii) a grouping strategy that integrates PSUM quantization (PSQ) and APSQ with floating-point regularization techniques to enhance accuracy while minimizing accuracy loss and reducing energy cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The embodiments of the present invention are described in more detail below with reference to the accompanying drawings, in which:

[0020] Figure 1A Demonstrated a high-precision (FP32 or INT32) piecewise linear approximation (PLA) implementation in hardware based on a lookup table (LUT) without the need for input quantization.

[0021] Figure 1B The proposed low-precision (INT8 or INT16) piecewise linear approximation (PLA) based on a lookup table (LUT) and a hardware implementation combined with input quantization were exhibited;

[0022] Figure 1C Provides a schematic diagram of the operating mechanism of the improved approximation method and architecture of the present invention;

[0023] Figure 2 Describe the specific implementation steps of the GQA-LUT algorithm;

[0024] Figure 3 Characterize the accuracy of the piecewise linear approximation at various scaling factors;

[0025] Figure 4 Visualize the error of the piecewise linear approximation caused by quantization;

[0026] Figure 5 Demonstrate rounding mutation algorithm;

[0027] Figure 6 Analyze the energy performance of the Softmax engine with near-memory computation;

[0028] Figure 7 Analyze the energy performance of folded LayerNorm with near-memory computation;

[0029] Figure 8 Demonstrate the architecture of the Special Purpose Computing Unit (SPU);

[0030] Figure 9 A schematic diagram showing the DNN accelerator for analysis of the present invention;

[0031] Figure 10A Demonstrates an example solution for point-wise convolution with a tile-based computation scheme;

[0032] Figure 10B and Figure 10C Schematic diagrams showing INT-8 partial and quantization and additive partial and quantization, respectively;

[0033] Figure 11 Demonstrate grouping strategy algorithms;

[0034] Figure 12A and 12B Schematic diagram showing the accuracy and additional quantization time across different group sizes under Segformer-B0 and EfficientViT-B0, respectively;

[0035] Figure 12C Shows a table comparing PSUM quantization methods;

[0036] Figure 13A and 13B Analytical plots of normalized energy across different data flows, accuracy, fractional and quantization methods, and group sizes for Segformer-B0 and EfficientViT-B0 under input static (IS) and weight static (WS), respectively;

[0037] Figure 14A and Figure 14B An analytical plot is presented, which shows that as the Bo constraint is tightened, the normalized energy of larger gs increases, which is more significant in the IS data stream;

[0038] Figure 15 A schematic diagram of a system according to an embodiment of the present invention is shown, wherein the system adopts an improved approximation architecture and executes a piecewise linear approximation method to achieve an efficient converter acceleration procedure. DETAILED DESCRIPTION

[0039] In the following description, the system and method are described as preferred examples, and the system and method are used for the genetic LUT approximation algorithm (GQA-LUT) for INT8 quantization and near-memory architecture. It will be clear to those skilled in the art that modifications, including additions and / or substitutions, can be made without departing from the scope and spirit of the invention. Specific details may be omitted so as not to obscure the invention; however, this disclosure is written to enable those skilled in the art to practice the teachings herein without undue experimentation. In addition, in this disclosure or herein, algorithm codes related to the technical content are shown. These algorithm codes are presented in pseudocode form, and are intended to enable those skilled in the art to understand and practice the technical solutions described herein after reading them.

[0040] Advantages or improvements of the present invention over the prior art:

[0041] The proposed GQA-LUT is compared in detail with the state-of-the-art PLA method, NN-LUT. The accuracy of these methods is initially examined at the operator level across a range of nonlinear functions. Subsequently, the fine-tuning accuracy of two hybrid transformer and CNN models, Segformer and EfficientViT, is evaluated, specifically in the domain of semantic segmentation tasks under integer-only quantization. Furthermore, LUT-based PLA units with varying precision are implemented using Verilog HDL, and their hardware performance, including metrics such as area and power consumption, is benchmarked.

[0042] Advantages or improvements of the present invention over the prior art: (a) Improvement in operator-level accuracy:

[0043] In this paper, five nonlinear functions commonly used in transformer architectures and their variants are explored, namely: GELU, EXP, HSWISH, DIV and RSQRT. In addition, the NN-LUT has been re-implemented based on the training method, and the slope, intercept and breakpoints are directly converted to match the accuracy level of the GQA-LUT. In order to thoroughly evaluate the accuracy at the operator level, the impact of the scaling factor with increasing step size on the dequantized INT8 data is analyzed in detail. Since only GELU, EXP and HSWISH are affected by the scaling factor S, the results show that the GQA-LUT with rounding variation (RM) can show superior performance compared to the NN-LUT at different S values.

[0044] Advantages or improvements of the present invention over the prior art - (b) Improvement in accuracy of fine-tuning model:

[0045] In this paper, fine-tuning accuracy is further evaluated on the Cityscapes dataset, a semantic segmentation benchmark with pixel-level annotations across 19 categories. We focus on two transformer models: Segformer-B0 at 1024×1024 resolution using nonlinear operators such as EXP and GELU, and EfficientViT-B0 at 1920×1024 resolution designed for edge devices using HSWISH and DIV operators. Both models are INT8 quantized using the LSQ method, forming the baseline. Evaluated using the mean intersection over union (mIoU) metric, the results in Tables 1 and 2 show that GQA-LUT with RM has the least impact on fine-tuning accuracy, with a loss of only 0.07% for Segformer-B0 and 0.02% for EfficientViT-B0, outperforming the NN-LUT method with 1.07% and 0.88% improvements in loss, respectively. Table 1: Segformer-B0 fine-tuning mIoU on Cityscapes replace* NN-LUT GQA-LUT without RM GQA-LUT with RM none 74.60％ 74.60％ 74.60％ EXP only 73.88％ 74.31％ 74.59％ GELU only 73.61％ 74.24％ 74.57％ DIV only 74.15％ 74.37％ 74.58％ RSQRT only 73.94％ 74.17％ 74.51％ Together 73.46％ 74.28％ 74.53％ *: Non-linear function replaced by 8-entry PLA Table 2: EfficientViT fine-tuned mIoU on Cityscapes *: Non-linear function replaced by 8-entry PLA

[0046] Advantages or improvements of the present invention over the prior art - (c) Advantages of hardware performance

[0047] In order to further verify the necessity and advantages of GQA-LUT with low bit precision (INT8), VerilogHDL is used to design Figure 1A and Figure 1B The two types of hardware units are depicted. The area and power consumption metrics for each LUT-based unit were obtained by synthesizing it using the Synopsys Design Compiler using 28-nm technology. To balance the comparison, the operating frequency was normalized to 500MHz on all units. The results presented in Table 3 show that the 8-entry INT8 LUT-based PLA occupies only 961um 2, demonstrating significant area reductions of 81.3% and 81.7% compared to high-precision FP32 and INT32 units, respectively. In terms of power efficiency, the INT8 configuration utilizes only 0.4mW, resulting in substantial power savings of 80.2% of FP32 and 79.3% of INT32. Furthermore, increasing the LUT size to 16 entries results in an approximately 1.71x increase in area and a 1.95x increase in power consumption compared to the 8-entry INT8 setting. This further highlights the importance of choosing fewer entries and lower precision configurations for piecewise linear (PWL) functions when following pure integer quantization, thereby effectively optimizing hardware resource utilization and improving energy efficiency. Table 3: Hardware costs for 28-nm technology

[0048] With reference to the relative energy consumption of MAC and different types of SRAM, the advantages of applying near-memory computing techniques to Softmax and LayerNorm using the GQA-LUT approach are discussed.

[0049] Figure 6 The normalized energy decomposition of the original Softmax pair computation energy (OP), GBuf read / write energy, and LBuf read / write energy is shown. Near-memory computation techniques are beneficial for Softmax because it involves multiple rounds of buffer reads and writes. Figure 6 It can be concluded that the relative energy consumption of Softmax is reduced by 43.9% when LBuf is applied.

[0050] like Figure 7 As shown in

[15] , LayerNorm with near-memory computation and further with folding arithmetic shows that the energy savings from applying near-memory computation is 35.4%. In addition, LayerNorm folding, which reduces computational complexity and memory access burden by fusing element-wise affine with general convolution operations, can achieve a substantial energy savings of 55.4%.

[0051] The following specific examples are provided to illustrate embodiments of the present invention.

[0052] Example 1: LUT-based PLA using genetic algorithm (GQA-LUT) and its hardware implementation.

[0053] Figure 1A Demonstrated a high-precision (FP32 or INT32) piecewise linear approximation (PLA) implementation in hardware based on a lookup table (LUT) without the need for input quantization. Figure 1B The proposed low-precision (INT8 or INT16) piecewise linear approximation (PLA) based on a lookup table (LUT) and its implementation in hardware combined with input quantization were exhibited; and Figure 1C A schematic diagram of the operating mechanism of the improved approximation method and architecture of the present invention is provided.

[0054] Using PLAs for a series of nonlinear operations, combined with storing parameters in LUTs, has become a widely accepted strategy for improving hardware efficiency. This approach has recently been validated and supported by several important studies. A PLA based on an N-entry LUT can be formulated as follows:

[0055] like Figure 1A As shown in , the parameters for an N-entry PLA are stored in a LUT. This setup allows for high accuracy due to the FP / INT32-based LUT storage and the superior representation of input data. However, challenges may arise in scenarios involving quantization with low bit precision.

[0056] Quantization has become an advantageous method for reducing computational and memory overhead when deployed on a chip, as it facilitates reasoning programs using low-bit representations. The quantization function is defined as follows:

[0057] where Q k (·) represents a k-bit quantization function that compresses high-precision input data x into low-bit INT-k by scaling factor α. Specifically, for n and Q p Defined signed and unsigned data, Q k (x) Constrained in [-2 k-1 ,2 k -1 -1] or [0,2 k -1]. After dequantization By changing Q k (x) is rescaled to retrieve α determined using a min-max technique or a learnable alternative. In general, nonlinear functions are linearly inseparable, such as exp(α·Q k (x))≠α·ep(Q k (x)). The approximation can be applied directly to x, but lacks generality due to different data flows. For the PLA method, it can be obtained by PLA(α·Q k (x)K=α·PLA(Q k The inherent property of (x)K brings separability to nonlinear functions, while current work based on PLA only focuses on optimizing PLA(α·Q k (x)K, the arithmetic process is still in high-precision FP / INT format. PLA provides a unique advantage, allowing direct operation on the quantized data Qk (x) Perform the PWL operation while separating the scaling factor α.

[0058] Additionally, the scaling factor is set to a power of 2. This is achieved by rounding the logarithm of the learnable parameter α to its nearest integer. Then, the The scaling factor is derived. This power-of-two adjustment of α is designed to simplify the PLA process when quantization is required, because if the scaling factor is a power of two, all operations can be completed by the shifter in hardware. Based on the above analysis, Figure 1B A PLA based on quantization-aware LUT is proposed in

[15] .

[0059] However, the performance of LUT-based PLA is highly dependent on the breakpoint p i , slope k i and intercept b i To better consider the quantization effect that previous neural network-based methods failed to achieve, Figure 2 The proposed genetic PLA algorithm is described in

[15] . The core concept involves creating a population of individuals representing the breakpoints p of different piecewise linear functions. i . These individuals undergo a random process of crossover and mutation to promote diversity and discover new solutions. Crossover involves the exchange of segments between pairs, while mutation incorporates a normal distribution of noise to introduce variability. The individuals that exhibit the highest fitness, identified by the lowest mean squared error (MSE) when approximating the nonlinear function f(·), are then selected. This process embodied by the GQA-LUT reflects the principle of natural selection, where the MSE acts as the selection criterion. In order to facilitate storage and computation in a compact INT format, a direct method of converting the best set of slopes and intercepts from FP32 to FXP values ​​using a specified decimal bit width λ is adopted. At the same time, the breakpoints are quantized as depicted in equation (2). For a given f(·), the selection of λ is guided by searching the interval [Rn,Rp]. When Figure 3 As shown in Figure 5, when comparing the accuracy of an 8-entry GQA-LUT and a NN-LUT using the same FXP transformation technique used to approximate the GELU, it can be observed that the GQA-LUT outperforms the NN-LUT at all scaling factors.

[0060] Example 2 - Mutation algorithm incorporating quantization effects into the above GQA-LUT.

[0061] A detailed analysis of the MSE of GQA-LUT indicates that its main challenge lies in managing large α values, such as Figure 3In particular, scaling factors less than 2^-2 account for more than 90% of the total error, highlighting the significant limitation in effectively handling large α. To understand why the approximation error is more pronounced at higher values ​​of α, the approximation curve of the exponential function (EXP) can be analyzed, as shown in Figure 4 As shown in Figure 2, when the breakpoint p is quantized to a specific integer value according to Equation (2), the resulting approximation error exhibits variability across different scaling factors α. Notably, at higher values ​​of α, the breakpoint tends to shift significantly, leading to a noticeable approximation shift—an effect we call breakpoint bias. On the other hand, lower α tends to produce minimal bias, effectively reducing the error. This insight highlights the limitations of using a direct FXP conversion for GQA-LUTs, especially at higher scaling factors, where breakpoint bias becomes significantly more significant.

[0062] To solve this challenging problem, Figure 5 The method called Rounding Mutation (RM) is described in detail. The RM technique conceptualizes the quantization of breakpoints as random mutation, where quantization occurs randomly over different factors α, affecting each element within a set of breakpoints in its evolutionary trajectory. In the context of GQA-LUT, the traditional variogram based on normally distributed noise is replaced with the proposed RM method, while still maintaining the direct FXP transformation of slope and intercept. Figure 3 As shown in , incorporating the RM strategy into the GQA-LUT significantly reduces the MSE for large α values. Although there is a slight increase in MSE for smaller α values, the error is initially low, so this increase is negligible and can be ignored in practice.

[0063] Example 3 - Using the above example for a near-memory computing hardware implementation of quantized Softmax and folded-based LayerNorm

[0064] Since Softmax and LayerNorm have similar memory access rules, such as word-based memory reading and writing, they can be configured together as a special computing unit (SPU). The SPU can be embedded in an AI accelerator. Figure 8 As shown in the , the SPU uses the instruction set and utilizes a decoder structure to determine operations and memory addresses. The Softmax and LayerNorm submodules share a local buffer of less than 64KB for near-memory computations, thereby reducing memory access energy and latency due to the copy memory accesses required in both operations. Thanks to the configurability of the GQA-LUT, these two operations can share multiple INT8 GQA-LUT modules and are used for piecewise linear approximation (PLA) of exponential, square root, and reciprocal.

[0065] In addition, LayerNorm folding is proposed, which fuses the LayerNorm parameters with the weights and biases of consecutive convolutional layers to reduce memory and computational overhead. For example, consider a point-by-point convolutional layer connected after a LayerNorm layer, and the input of the convolutional layer is: x ik =l ik γ k +β k , (3)

[0066] Where i represents the i-th token and k represents the k-th input channel among the total C input channels. l is the initial LayerNorm without element-wise affine transformation. γ k and β k are the weights and biases of the affine transformation. The output of the convolutional layer is:

[0067] Where j represents the jth filter of the convolutional layer after LayerNorm. Then, replace x with the LayerNorm parameter ik to obtain:

[0068] The equation can be organized as:

[0069] Therefore, the folding weights of the convolution are inferred to be:

[0070] And, the folding bias is:

[0071] LayerNorm folding simplifies the element-by-element affine transformation that requires weight buffer access and MAC operations. As a result, the SPU no longer requires IO for weight buffer access, alleviating placement burden and timing issues.

[0072] To accelerate Softmax or LayerNorm, the SPU first receives instructions from the host processor or stage controller and then executes them according to the data flow controller. Only the first memory read access and the final writeback process require access to GBufs; the rest are performed in the SPU using LBufs. All calculations are performed by shared arithmetic to maximize resource reuse of the multiplexer.

[0073] Example 4.1: Energy Efficiency Analysis

[0074] In the field of DNN accelerators, a key factor is energy efficiency in terms of operations / watt, which is significantly affected by accuracy, layer size, data flow and architecture. Figure 9 The method shown in [1] examines the energy efficiency of a typical DNN accelerator system. This system consists of a multiply-accumulate (MAC) array, on-chip SRAM, off-chip DRAM, and a top-level controller. The MAC array is based on P o 、P ci and P co The total energy cost E is defined as follows: total yes: E total =N d ·E dram +N s ·E sram +N m ·E mac (9)

[0075] Among them E dram 、E sram and E mac N represents the energy cost for each access to DRAM and SRAM and for a single MAC operation, respectively. d 、N s and N m Corresponds to the total number of accesses to DRAM and SRAM, and the total number of MAC operations used. The energy cost value is taken from E dram =162.5pJ / byte, E sram =12.5pJ / byte and E mac = 0.3 pJ / MAC for subsequent analysis. Since the number of MACs for a specific DNN is constant, the energy efficiency depends largely on E dram / sram .

[0076] In the analysis framework based on some known research, weights, activations, and PSUM are all stored in 16-bit fixed-point format. However, PSUM may require higher precision, such as INT32 in some DNN accelerators that only use integer quantization. To accommodate this change, the framework is modified by introducing the following precision factor:

[0077] The total number of accesses to DRAM / SRAM for ifmap, weight, PSUM and ofmap is defined as N i d / s 、N w d / s 、N p d / s and N o d / s , where S i 、S w and S o Indicates the size of ifmap, weights, and ofmap. β is the ratio of PSUM accuracy to the accuracy of feature maps / weights. For example, if PSUM is in INT32 for an INT-8 DNN, β will be 4. In this paper, the focus is only on INT-8 DNNs with WS or IS, because OS does not encounter issues with PSUM accuracy.

[0078] Regarding weight quiescence, the WS-based accelerator loads the ifmap tile to update the PSUM by reusing the Pci×Pco kernel weights, and generates the ofmap tile after completing the PSUM accumulation. Accordingly, the components of Ns can be derived as:

[0079] Where Ci and Co represent the dimensions of the input channel and output channel, Bi and Bo indicate the capacity of ifmap and ofmap in bytes. The height and width of ofmap are denoted as H o and W o In addition, the input tile size S is scaled up based on the output tile, kernel, and stride ～ i . S ～ p is calculated as β×P o ×P co The size of the tile PSUM, where P oh and P ow is the height and width of the output / PSUM tile, affected by P o =P oh ×P ow constraint.

[0080] If Bi or Bo is exceeded, an additional read to both SRAM and DRAM occurs, and Ni / w / p / od is defined as:

[0081] In contrast, regarding input quiescence, the IS-based accelerator keeps the input tiles quiescent within the registers of the MAC array, thereby facilitating their reuse by weights to update the PSUM in the output buffer. Similarly, Ni / w / p / os is summarized as follows:

[0082] The weight buffer size is denoted by Bw, and the height and width of the enlarged ifmap are Hi and Wi respectively. In addition, the ifmap tiles are shaped by Pih and Piw following Pi=Pih×Piw. It is worth noting that the PSUM of IS is larger than that of WS because the ifmap tiles need to be reused along the entire Co by weights. Therefore, S ～ p scales with Co / Pco. Ni / w / p / op for DRAM is derived as:

[0083] Example 4.2: Integer-only quantization

[0084] Quantization has become an effective strategy to alleviate computational and memory overhead, particularly beneficial for deployment on hardware accelerators, and can be expressed as:

[0085] where Q k (·) represents the k-bit quantization function that compresses high-precision input data into low-bit INT-k by scaling factor α. Specifically, for n and Q p Defined signed and unsigned data, Q k (x) constrain xα to be within [-2 k-1 ,2 k-1 -1] or [0,2 k -1]. After dequantization x ～ By changing Q k (x) rescaling is retrieved using α determined using a min-max technique or a learnable alternative that is more favorable for its effectiveness. Specifically, quantization can be categorized into integer-only and analog variants, characterized in that the former only contains low-cost integer arithmetic, while the latter relies heavily on high-precision dequantized data. In this work, integer-only quantization is chosen, where α for both weights and activations / PSUMs is obtained via LSQ and forced to binary and power-of-two formats, respectively, via a straight-through estimator (STE).

[0086] Example 4.3: Partial Sum Quantization

[0087] Due to computational limitations, tile-based computing (TBC) is the dominant approach in DNN accelerators. This technique often involves additive interaction with PSUM to produce the final output tile. i 、T w 、T p and T o The tiles defined as ifmap, weight, PSUM and ofmap respectively can be found in Figure 10A An example of point-wise convolution with TBC is shown in Fig. o is derived from the accumulation of multiple PSUMs generated by multiplying the elements in Ti and Tw. Therefore, TBC can be formulated as follows:

[0088] where n ( Represents PSUM tile T pi The number of, where 0≤i≤n p -1.

[0089] Although TBC ensures computational efficiency, the necessity of high precision in PSUM may still introduce overhead in some CIM designs. Therefore, they all proposed PSUM quantization (PSQ), such as Figure 10B It is described and summarized as follows:

[0090] Each Q k (·) and α are annotated with superscripts related to their PSUM. APj represents the jth PSUM tile T pj The above equation and Figure 10B shows that, due to additive addition, AP j The accuracy is higher than Q k (T p ) accuracy, which results in a significantly higher energy cost for PSUM.

[0091] Example 5.1: Additive Parts and Quantization

[0092] Given that recent PSQ methods fail to solve the energy problem associated with high-precision PSUMs, which arises from the need to accumulate several dequantized PSUMs to generate a new PSUM, which inevitably leads to frequent regeneration of high-precision PSUMs, the accumulation process should precede quantization in order to achieve low-bit memory access across all PSUMs. Therefore, this paper proposes a novel additive PSUM quantization method called APSQ, which is described in detail as follows: where 1≤i≤n p (17)

[0093] Within APSQ, each Api is recursively determined by summing the Qik applied to Tpi with the previous additive PSUMAP i-1 for 1 ≤ i < np and is depicted in Figure 10C . The initial additive PSUMAP0 is set to Q 0 k (T p0 ). The output tile T o is derived upon completion of quantization of the final PSUM. Thus, T o can be concisely represented as APn p-1 . Notably, even though promoting low - memory access of PSUM, APSQ subjects the early PSUM to multiple quantizations cyclically. Taking Tp0 as an example, it will encounter np quantizers during the accumulation process and eventually become T o .

[0094] Example 5.2: Exploration Strategy

[0095] As previously discussed, participating in multiple quantizations usually leads to poor performance. To address this issue, reducing the quantization frequency becomes crucial as it can reduce the additional rounding errors of the early PSUM. Based on this insight, a grouping strategy that combines APSQ and PSQ is proposed and detailed in the algorithm in Figure 11 .

[0096] The described strategy begins by partitioning the set of PSUM tiles into groups of size gs. Within each group, APSQ is applied only once, focusing on the accumulation of the current input PSUM tile added to the gs de - quantized PSUM tiles via PSQ from the last group. The grouping strategy keeps the total memory read and write operations unchanged, but merges the reads of PSQ into the input of APSQ for the next group, resulting in a modest increase in the PSUM storage size. However, due to the small size of the PSUM tiles, the increase is usually negligible. Additionally, the additional quantization count is formulated as:

[0097] where [condition] is an indicator function that equals 1 when the condition is satisfied and 0 otherwise. Due to the grouping strategy, an increase in gs can reduce the total count of additional quantizations throughout the process.

[0098] Example 5.3: Floating - point Regularization

[0099] To further improve the accuracy of the above method, a floating-point regularization (FPR) strategy is introduced. The L2 norm can be calculated as a measure of the difference between the floating-point inference results and the output obtained from the APSQ / PSQ grouping strategy. During the training phase, the FPR is integrated to guide APSQ and PSQ to align with the accurate floating-point results, which can be formulated as:

[0100] in and Denote the i-th output tile determined by floating-point calculation and grouping strategy respectively. The symbol no denotes the total number of output tiles, and ∥·∥2 denotes the calculation of L2 norm. FPR aims to align the quantized PSUM closely with the PSUM in floating point. Moreover, the popular technique of using floating-point models as teachers in quantization, namely knowledge distillation (KD), is utilized. The total loss is defined as L=L TASK +λ k L KD +λ r L FPR , where λ k and λ r As the weighting factor of KD and FPR loss, and L TSAK Represents the loss for a certain task, such as the cross-entropy loss in semantic segmentation.

[0101] Example 6.1: Experimental Setup

[0102] In this section, we focus on the challenging task of semantic segmentation, specifically targeting the Cityscapes dataset, which depicts urban scenes across 19 categories and contains 2,975 images for training, 500 images for validation, and 1,525 images for testing. In addition, the presented research includes two models: Segformer-B0, which adopts a traditional transformer architecture with vanilla attention; and EfficientViT-B0, which is a hybrid model that combines a large number of convolutions with linear attention. Only integer quantization-aware training (QAT) is initially combined with a full-precision teacher model that performs knowledge distillation (KD) to obtain a baseline model. Subsequently, a second stage of KD is performed, in which the previously quantized network combined with APSQ or PSQ replaces the student model. Notably, the standard mean intersection over union (mIoU) is used as the evaluation metric in the experiments.

[0103] like Figure 9The setup of the analytical DNN accelerator depicted in

[15] begins with the allocation of memory buffers: 128KB each for the input feature map (ifmap) and output feature map (ofmap), and 64KB for the weight buffer, assuming that off-chip DRAM is sufficient. In the configuration of the MAC array, parallelization is organized as Po=16, Pci=8, and Pco=8, which determines the number of accesses to SRAM and DRAM, thus affecting the energy cost. Pci and Pco also determine the number and size of PSUMs in the TBC, respectively. Figure 9 As shown, in some CIM designs it may be much larger, for example, Pci / co=128.

[0104] Example 6.2: Accuracy Analysis

[0105] As mentioned above, applying APSQ to all PSUMs is the same as the grouping strategy with gs=1, which significantly degrades performance due to the huge additional quantization for each PSUM. To explore this hypothesis, we perform the following on all weights, activations, and PSUMs under INT-8 quantization: Figure 12A and Figure 12B Accuracy evaluation for different gs sizes is shown. The results clearly show that at gs=1, the accuracy is greatly affected, indicating that APSQ alone is insufficient. Furthermore, gradually increasing gs restores the accuracy while reducing the number of additional quantization counts.

[0106] Thus, there is a significant correlation between group size and the degree of additional quantization of PSUM, with larger group sizes being synonymous with superior performance. Figure 12C Table 4 further evaluates model accuracy at various PSUM bit widths, with gs = 6 fixed. Results demonstrate that APSQ can effectively reduce the precision factor β and, when combined with FPR techniques, enhances performance. Compared to the baselines of Segformer-B0 and EfficientViT-B0, when all convolutions, attention, and matrix multiplications are equipped with INT8 APSQ under the FPR strategy, mIoU only decreases by 0.44% and 0.20%, respectively, demonstrating minimal impact on accuracy.

[0107] Example 6.3: Energy Analysis

[0108] Continuing with the previous analysis of accuracy, an evaluation of the energy costs associated with different PSUM accuracies and gs is performed. Figure 13A and Figure 13BAs depicted in Figure 2, there is a discernible trend that lower PSUM precision is associated with reduced energy cost, and the energy savings for precision levels below 8 bits are very marginal, indicating an inflection point where further precision reduction does not yield significant energy benefits. Compared to INT8 APSQ, both Segformer-B0 and EfficientViT-B0 exhibit significantly higher energy costs when equipped with PSQ, 1.59x and 1.47x higher in IS and 1.81x and 1.43x higher in WS, respectively. Therefore, the energy savings achieved with INT8 APSQ are 30-45%. The superior performance of INT8 APSQ is attributed to its ability to store partial sums (PSUMs) in a low-bit format, a feature not present in PSQ, further demonstrating that INT8 precision is sufficient for APSQ's needs. Figure 13A and Figure 13B The results in

[15] also indicate that changes in gs have a negligible impact on energy consumption. This is consistent with the insights from the previous section, which include that increasing gs does not proportionally increase memory read / write operations, which are closely related to energy consumption. Therefore, in the context of APSQ, the accuracy of PSUM primarily determines energy usage, while, as previously discussed, gs has a more significant impact on accuracy, thus positioning it as a parameter for exploration.

[0109] Example 6.4: Design Space Exploration

[0110] Given that INT8 quantization of PSUM is sufficient for APSQ with respect to energy efficiency for a specific on-chip SRAM allocation, the accuracy of INT8 APSQ is therefore heavily dependent on gs. Given the variability of on-chip SRAM sizes across different DNN accelerators, we explored the correlation between gs and the SRAM size allocated for the ofmap buffer, as only the ofmap buffer affects PSUM. We conducted design space exploration (DSE) to evaluate the trade-offs, focusing on energy consumption, model accuracy, and data flow for various SRAM sizes gs. Figure 14A and Figure 14BAnalysis plots are presented, showing that as the Bo constraint is tightened, the normalized energy for larger gs increases, especially more significantly in the IS dataflow. This increase is mainly due to the IS dataflow's need to reuse ifmap tiles across the entire output channel Co, which exceeds the reuse range of the WS dataflow. Furthermore, the selection of the optimal gs can be guided by energy cost. For example, choosing a gs of 4 maintains accuracy while avoiding the additional energy overhead of IS-based DNN accelerators, especially for Segformer-B0 with a Bo of 64KB and EfficientViT-B0 with a Bo of 32KB. Such DSE is crucial for designing DNN accelerators that utilize APSQ with a grouping strategy to optimize trade-offs.

[0111] In this disclosure, a refinement analysis framework is described to evaluate the energy efficiency of deep neural network (DNN) accelerators, with a particular focus on the impact of partial sum (PSUM) accuracy. In the analysis presented, limitations of previous PSQ methods are pointed out, in particular, their inability to implement low-bit storage of PSUM. To overcome this, in this invention, a new APSQ method is proposed, which is enhanced by a grouping strategy and floating-point regularization. Experimental results verify that there is a large degree of correlation between gs and additional PSUM quantization. In the semantic segmentation task, the implementation of INT8 APSQ is tested on the complex Cityscapes dataset, and the results show that the adopted method has almost no accuracy loss in operations such as convolution, attention calculation and matrix multiplication. Design space exploration (DSE) experiments further emphasize the importance of balancing energy consumption and accuracy under the constraints of hardware resources and data flow patterns, which provides a strategic framework for optimizing DNN accelerators using APSQ.

[0112] Figure 15 A schematic diagram of a system 100 according to an embodiment of the present invention is shown, wherein the system adopts an improved approximation architecture and executes a piecewise linear approximation method to achieve an efficient converter acceleration process.

[0113] The system includes a LUT-based approximation circuit 110, a quantization-aware enhancement module 120, a low-bit LUT-based piecewise linear approximation circuit 130, a memory module 140, an output module 150, an additive partial sum (PSUM) quantization module 200, an accuracy enhancement module 210, and a floating-point regularization (FPR) module 220. The system 100 receives input data and processes the input data through its components. In one embodiment, the input data includes a set of high-resolution images with semantic labels and a semantic dataset. In this regard, each high-resolution image can have multiple pixel-level classifications corresponding to predefined categories, and the input data will be processed through a semantic segmentation task. The semantic dataset can have multiple training, validation, and test images and can be configured to be used in model training, model optimization, and model evaluation.

[0114] The LUT-based approximation circuit 110 is configured to adaptively approximate a nonlinear operator using a genetic algorithm 112. The genetic algorithm 112 evaluates multiple candidate solutions to generate optimal approximation parameters. The LUT-based approximation circuit can convert at least one original floating-point operation into at least one fixed-point operation using the parameters, such that the LUT-based approximation circuit outputs a converted fixed-point representation of the original floating-point operation based on the approximation parameters. In one embodiment, the genetic algorithm 112 in the LUT-based approximation circuit 110 is configured to dynamically adapt to input data and the nonlinear operator by continuously refining the approximation parameters.

[0115] The quantization-aware enhancement module 120 is configured to execute a rounding mutation algorithm 122 that treats the conversion from floating point to integer as a mutation process. The quantization-aware enhancement module 120 can operate in conjunction with the LUT-based approximation circuit 110 by refining the fixed-point output from the LUT-based approximation circuit 110 by rounding the values ​​in a manner that minimizes quantization error while maintaining accuracy. Thus, the quantization-aware enhancement module 120 outputs a quantized integer representation of the transformed fixed-point operation. In one embodiment, the quantization-aware enhancement module 120 is further configured to execute the rounding mutation algorithm 122 to adapt to the characteristics of the input data.

[0116] The low-order LUT-based piecewise linear approximation circuit 130 is configured to store all approximation parameters in INT8 format, thereby reducing memory and computational overhead, wherein the low-order LUT-based piecewise linear approximation circuit receives the quantized integer representation from the quantization-aware enhancement module 120 and performs piecewise linear approximation. The low-order LUT-based piecewise linear approximation circuit 130 can output a low-order approximation of the nonlinear operation. The memory module 140 is then configured to receive the result of the low-order LUT-based piecewise linear approximation circuit 130 and store the low-order approximation of the nonlinear operation in the efficient storage INT8 format.

[0117] In one embodiment, the low-bit LUT-based piecewise linear approximation circuit 130 operates at multiple levels of granularity, allowing for selection of a coarse or fine approximation strategy based on computational requirements. In one embodiment, the low-bit LUT-based piecewise linear approximation circuit 130 may include a feedback model that dynamically adjusts approximation parameters based on runtime performance metrics, thereby achieving continuous minimization of approximation error without sacrificing computational efficiency.

[0118] Furthermore, the additive PSUM quantization module 200 works together with the accuracy enhancement module 210 and the FPR module 220 for quantization perceptual enhancement module 120 .

[0119] The additive PSUM quantization module 200 is configured to quantize each accumulated PSUM in the matrix multiplication (MatMul) from FP32 format to INT8 format for quantizing the perception enhancement module 120. The additive PSUM quantization module 200 can be further configured to selectively quantize the partial sums based on the magnitude of the partial sums using a dynamic threshold method. The accuracy enhancement module 210 is configured to implement a grouping strategy that combines traditional PSUM quantization (PSQ) with additive PSUM quantization (APSQ) to improve accuracy, wherein the results of the PSQ are stored in on-chip SRAM and are read and accumulated to perform APSQ. The FPR module 220 is configured to further refine the accuracy of the accuracy enhancement module 210, wherein a mean square error (MSE) loss can be established between the approximate MatMul results from APSQ / PSQ and the accurate results from the original floating point operation.

[0120] In short, the additive PSUM quantization module 200 quantizes the partial sums from the MatMul process, converting them from FP32 to INT8. The additive PSUM quantization module 200 works with the accuracy enhancement module 210 and the FPR module 220 to optimize the quantization process and maintain accuracy. The accuracy enhancement module 210 combines the PSQ and APSQ strategies stored in SRAM for further processing. The FPR module 220 then ensures the accuracy of the refined quantization process by using the MSE loss function between the approximate floating point result and the original floating point result. With this configuration, the additive PSUM quantization module 200, the accuracy enhancement module 210, and the FPR module 220 are connected to the other components in the quantization process.

[0121] In one embodiment, the quantization-aware enhancement module 120 is coupled to the additive PSUM quantization module 200. The quantization-aware enhancement module 120 provides comprehensive quantization support, including strategies for optimizing accuracy and minimizing errors during the quantization process. At the same time, the additive PSUM quantization module 200 focuses on quantizing the accumulated partial sums during MatMul, thereby converting high-precision data into a lower-precision format such as INT8. For efficient quantization, pre-processed inputs from the quantization-aware enhancement module 120 (e.g., quantization parameters, scaling factors, or grouped / processed data) may be required to share quantization parameters with the additive PSUM quantization module 200. For example, the additive PSUM quantization module 200 can calculate and provide scaling factors, thresholds, or quantization ranges that the additive PSUM quantization module 200 applies during its operation. In this regard, if the additive PSUM quantization module 200 detects quantization errors or inefficiencies during its operation (e.g., data overflow or underflow during partial sum quantization), this feedback can be returned to the quantization-aware enhancement module 120 to adjust parameters such as scaling factors or quantization ranges.

[0122] The system 100 may further include a fine-tuning framework module 300 , a fixed-point quantization-aware training module 310 , and a fine-tuning module 320 .

[0123] The fine-tuning framework module 300 is configured to perform an offline piecewise linear approximation using a set of target input scaling factors. The fine-tuning framework module 300 can be further configured to adaptively select a scaling factor for each nonlinear operator such that the scaling factor is optimized for transforming each nonlinear operator into a piecewise linear approximation, aligned with the fixed-point conversion process performed by the LUT-based approximation circuit 110.

[0124] The fixed-point quantization-aware training module 310 is configured to compress a pre-trained floating-point model by initializing the scaling factors through fixed-point quantization. The fixed-point quantization-aware training module 310 can utilize the quantization-aware enhancement module 120 to refine the scaling factors while maintaining accuracy during the quantization process.

[0125] The fine-tuning module 320 is configured to update the quantized model during a subsequent fine-tuning process. In this regard, the forward pass replaces each nonlinear operator with a piecewise linear approximation function based on the corresponding scaling factor, and the backward pass further refines the scaling factor through knowledge distillation.

[0126] In one embodiment, the LUT-based approximation circuit 110 can establish a bidirectional connection with the fine-tuning framework module 300, the fixed-point quantization-aware training module 310, and the fine-tuning module 320. For this connection, the input to the LUT-based approximation circuit 110 can include quantization scaling factors, piecewise linear approximation parameters, and optimized fixed-point data provided by the fine-tuning framework module 300, the fixed-point quantization-aware training module 310, and the fine-tuning module 320. In addition, the fine-tuning module 320 supplies fine-tuned LUT update parameters, such as scaling factors or segmentation configurations, to further enhance the circuit's approximation accuracy. In response, the LUT-based approximation circuit 110 outputs operational feedback (e.g., approximation error or data distribution) to the fine-tuning framework module 300, the fixed-point quantization-aware training module 310, and the fine-tuning module 320. This feedback allows these modules to iteratively refine scaling factors and approximation strategies.

[0127] The output module 150 is electrically coupled to the memory module 140 and configured to output a semantic segmentation map for each input data (e.g., input image). In one embodiment, the segmentation map may include a pixel-by-pixel classification of all pixels in the input image and output an evaluation result based on a mean intersection over union (mIoU) metric representing the accuracy of the semantic segmentation task.

[0128] System 100 may further include a convolution module 400 and a CNN module 410. Convolution module 400 is configured to perform a convolution operation on input data using a set of learned filters, wherein convolution module 410 utilizes a piecewise linear approximation method of LUT-based approximation circuit 110 to accelerate the convolution operation. CNN module 410 is configured to utilize the output from convolution module 400. In one embodiment, CNN module 410 is trained using a machine learning algorithm to perform tasks for image classification, object detection, or semantic segmentation.

[0129] In one embodiment, a near-memory computation engine 500 for Softmax and LayerNorm operations is provided. The near-memory computation engine 500 includes a near-memory architecture model 510, a piecewise linear approximation engine 520, and a LayerNorm folding module 530. The near-memory architecture model 510 is customized to reduce the data movement burden associated with Softmax and LayerNorm computations. The piecewise linear approximation engine 520 is implemented via the LUT-based approximation circuit 110 and is configured to utilize a lookup table and quantization method to minimize data transfer and enhance overall computational efficiency. The LayerNorm folding module 530 is configured to reduce the required parameters and computational steps.

[0130] Specifically, the near-memory architecture model 510 is connected to the LUT-based approximation circuit 110 to optimize data transfer, thereby reducing the data movement burden associated with the Softmax and LayerNorm operations. The piecewise linear approximation engine 520 can process these operations by utilizing piecewise linear approximations and is connected to the near-memory architecture model 510. In addition, the LayerNorm folding module 530 interacts with the LUT-based approximation circuit 110 to optimize the calculation of LayerNorm by reducing the required parameters. These components can work together or collaborate with each other to enhance the efficiency of the Softmax and LayerNorm operations, thereby minimizing the computational load and data transfer requirements. The connections between these components / modules may be bidirectional, allowing feedback from the approximation and folding processes to iteratively optimize the scaling factors and approximation parameters.

[0131] The functional units and modules of the apparatus and methods according to the embodiments disclosed herein can be implemented using a computing device, a computer processor, or an electronic circuit system, including but not limited to an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a microcontroller, and other programmable logic devices configured or programmed according to the teachings of the present disclosure. The computer instructions or software codes executed in the computing device, computer processor, or programmable logic device can be easily made by practitioners in the field of software or electronic technology based on the teachings of the present disclosure.

[0132] All or part of the method according to the embodiment may be executed in one or more computing devices including a server computer, a personal computer, a laptop computer, a mobile computing device (eg, a smartphone), and a tablet computer.

[0133] Embodiments may include computer storage media, transitory and non-transitory memory devices having stored therein computer instructions or software codes that can be used to program or configure a computing device, a computer processor, or an electronic circuit system to perform any of the processes of the present invention. The storage media, transitory and non-transitory memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray disks, DVDs, CD-ROMs and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of media or device suitable for storing instructions, code, and / or data.

[0134] Each of the functional units and modules according to various embodiments may also be implemented in a distributed computing environment and / or cloud computing environment, where all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0135] The foregoing description of the present invention has been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0136] The embodiment was chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A system using an improved approximation architecture that implements a piecewise linear approximation method to achieve efficient converter acceleration, characterized in that: The system comprises: a LUT-based approximation circuit configured to adaptively approximate a nonlinear operator using a genetic algorithm, wherein the genetic algorithm evaluates a plurality of candidate solutions to generate optimal approximation parameters, and the LUT-based approximation circuit converts at least one original floating-point operation to at least one fixed-point operation using the parameters, such that the LUT-based approximation circuit outputs a transformed fixed-point representation of the original floating-point operation based on the approximation parameters; a quantization-aware enhancement module configured to execute a rounding mutation algorithm that treats conversion from floating point to integer as a mutation process, wherein the quantization-aware enhancement module cooperates with the LUT-based approximation circuit to round a fixed-point output generated by the LUT-based approximation circuit to minimize quantization error while maintaining accuracy, thereby causing the quantization-aware enhancement module to output a quantized integer representation of the transformed fixed-point operation; a low-bit LUT-based piecewise linear approximation circuit configured to store all approximation parameters in an INT8 format, thereby reducing memory and computational overhead, wherein the low-bit LUT-based piecewise linear approximation circuit receives the quantized integer representation from the quantization-aware enhancement module and performs piecewise linear approximation such that the low-bit LUT-based piecewise linear approximation circuit outputs a low-bit approximation of the nonlinear operation; and A memory module is configured to store the low-bit approximation of the nonlinear operation in an INT8 format for efficient storage.

2. The system according to claim 1, wherein: Further including: an additive partial sum (PSUM) quantization module configured to quantize each accumulated PSUM in a matrix multiplication (MatMul) from an FP32 format to an INT8 format for the quantization-aware enhancement module; an accuracy enhancement module configured to implement a grouping strategy that combines traditional PSUM quantization (PSQ) with additive PSUM quantization (APSQ) to improve accuracy, wherein results of the PSQ are stored in an on-chip SRAM and read and accumulated to perform the APSQ; and A floating point regularization (FPR) module is configured to further refine the accuracy of the accuracy enhancement module, wherein a mean square error (MSE) loss is established between the approximate MatMul result from APSQ / PSQ and the accurate result from the original floating point operation.

3. The system according to claim 2, characterized in that The additive PSUM quantization module is further configured to employ a dynamic threshold method to selectively quantize the partial sums based on their magnitudes.

4. The system according to claim 1, wherein: Further including: a fine-tuning framework module configured to perform an offline piecewise linear approximation using a set of target input scaling factors, wherein the fine-tuning framework module is further configured to adaptively select a scaling factor for each nonlinear operator such that the scaling factor is optimized for transforming each nonlinear operator into a piecewise linear approximation in alignment with a fixed-point conversion process performed by the LUT-based approximation circuit; a fixed-point quantization-aware training module configured to compress a pre-trained floating-point model by initializing the scaling factor via fixed-point quantization, wherein the fixed-point quantization-aware training module utilizes the quantization-aware enhancement module to refine the scaling factor while maintaining accuracy during the quantization process; as well as A fine-tuning module is configured to update the quantized model during a subsequent fine-tuning process, wherein a forward pass replaces each nonlinear operator with a piecewise linear approximation function based on a corresponding scaling factor, and a backward pass further refines the scaling factor through knowledge distillation.

5. The system according to claim 1, wherein: The genetic algorithm in the LUT-based approximation circuit is configured to dynamically adapt to input data and nonlinear operators by continuously refining the approximation parameters.

6. The system according to claim 5, characterized in that The quantization-aware enhancement module is further configured to perform a rounding mutation algorithm to adapt to characteristics of input data.

7. The system according to claim 6, characterized in that The input data includes: a set of high-resolution images with semantic labels, wherein each high-resolution image has a plurality of pixel-level classifications corresponding to predefined categories, and the input data is processed via a semantic segmentation task; and A semantic dataset that has multiple training images, validation images, and test images and is configured for use in model training, model optimization, and model evaluation.

8. The system according to claim 7, characterized in that Further including: An output module is electrically coupled to the memory module and configured to output a semantic segmentation map for each input image, wherein the segmentation map contains a pixel-by-pixel classification of all pixels in the input image, and output an evaluation result based on a mean intersection over union (mIoU) metric to characterize the accuracy of the semantic segmentation task.

9. The system according to claim 8, characterized in that Further including: a convolution module configured to perform a convolution operation on the input data using a set of learned filters, wherein the convolution module utilizes the piecewise linear approximation method of the LUT-based approximation circuit to accelerate the convolution operation; A convolutional neural network (CNN) module configured to utilize output from the convolution module, wherein the CNN module is trained using a machine learning algorithm to perform tasks for image classification, object detection, or semantic segmentation.

10. The system according to claim 8, wherein: The low-bit LUT-based piecewise linear approximation circuit operates at multiple levels of granularity, allowing selection of a coarse or fine approximation strategy based on computational requirements.

11. The system according to claim 10, wherein: The low-bit LUT-based piecewise linear approximation circuit includes a feedback model that dynamically adjusts the approximation parameters based on runtime performance metrics.

12. A near-memory computing engine for Softmax and LayerNorm operations, characterized in that: include: A near-memory architecture model, tailored to reduce the data movement burden associated with Softmax and LayerNorm computations; A piecewise linear approximation engine implemented via the LUT-based approximation circuit according to claim 1 and configured to utilize a lookup table and quantization method to minimize data transfer and enhance overall computational efficiency; and LayerNorm folding module, which is configured to reduce the required parameters and computational steps.

Images