Hybrid-precision-based storage-computing circuit, data processing method and chip

Abstract

This invention relates to the field of integrated circuits and discloses a mixed-precision in-memory computing circuit, data processing method, and chip. The invention includes: a storage array, an index memory, and a multiply-accumulate calculation circuit. The storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format; wherein the bit width of the first precision format is greater than that of the second precision format; the storage locations of the storage array are divided according to the bit width of the second precision format; the first data element occupies multiple consecutive storage locations; the read bit width of the storage array is an integer multiple of the bit width of the first precision format; the index memory stores the position index information of the first data element in the storage array; the multiply-accumulate calculation circuit is used to determine the precision format corresponding to the data read from the storage array based on the position index information, and performs a multiply-accumulate operation corresponding to the precision format, outputting the calculation result. This solves the compatibility problem between mixed-precision data storage and computing circuits.

Description

Technical Field

[0001] This invention relates to the field of integrated circuits, and particularly to in-memory computing circuits, data processing methods, and chips based on mixed precision. Background Technology

[0002] With the widespread application of large-scale language models in the field of natural language processing, the large-scale matrix multiplication and accumulation operations involved in their inference process place higher demands on the computational efficiency and energy efficiency of hardware. In-memory computing technology, by integrating computational logic near the storage unit and reducing the frequent transfer of data between the processor and memory, has become an important technical direction for improving the energy efficiency of deep neural network inference.

[0003] In practical applications, model weight data often contains data elements with varying precision requirements. For example, some data elements that have a critical impact on inference accuracy need to be retained in a higher-precision format, while other data elements can be quantized in a lower-precision format to save storage space. However, current in-memory computing architectures face compatibility issues between storage formats and computing circuits when processing such mixed-precision data. Furthermore, the hardware overhead and computational latency during multiplication-accumulation operations need further optimization. Summary of the Invention

[0004] The purpose of this invention is to provide a mixed-precision in-memory computing circuit, data processing method, and chip to solve the problems of poor compatibility between storage format and computing circuit, high hardware overhead, and high computational latency in related technologies when processing mixed-precision data.

[0005] To address the aforementioned technical problems, this invention provides a mixed-precision in-memory computing circuit, comprising: a storage array, an index memory, and a multiply-accumulate calculation circuit electrically connected to the storage array and the index memory. The storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format; wherein the bit width of the first precision format is greater than the bit width of the second precision format; the storage locations in the storage array are divided according to the bit width of the second precision format; the first data element occupies multiple consecutive storage locations; the read bit width of the storage array is an integer multiple of the bit width of the first precision format; the index memory stores the position index information of the first data element in the storage array; the multiply-accumulate calculation circuit is used to determine the precision format corresponding to the data read from the storage array based on the position index information, perform a multiply-accumulate operation corresponding to the determined precision format, and output the calculation result.

[0006] The present invention also provides a data processing method applied to the above-mentioned mixed-precision in-memory circuit, comprising: storing data in a storage array; wherein the storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format, the bit width of the first precision format being greater than the bit width of the second precision format; the storage locations in the storage array being divided according to the bit width of the second precision format; the first data element occupying multiple consecutive storage locations; the read bit width of the storage array being an integer multiple of the bit width of the first precision format; storing the position index information of the first data element in the storage array in an index memory; reading data from the storage array and reading the corresponding position index information from the index memory, determining the precision format corresponding to the read data according to the position index information, performing a multiplication-accumulation operation corresponding to the determined precision format on the read data and the activation value, and outputting the operation result.

[0007] The present invention also provides a chip including the mixed-precision in-memory circuit described above.

[0008] In this invention, by uniformly dividing the storage locations in the storage array according to the bit width of the second precision format (i.e., the lower precision format), the first data element (i.e., the higher precision data element) can occupy multiple consecutive storage locations. This utilizes the storage space originally allocated to adjacent second data elements to accommodate the larger bit-width first data element. Without introducing additional coordinate indices or variable-length storage formats, the high-precision data element is naturally embedded into a storage structure with the low-precision format as its basic granularity. This maintains the regularity and continuity of storage access and avoids memory fragmentation caused by the bit width differences of mixed-precision data. Since the second data elements originally corresponding to the adjacent storage locations occupied by the first data element contribute relatively little to the overall computational result, exchanging the storage space of these low-sensitivity data elements for the high-precision preservation of high-sensitivity data elements effectively improves the overall computational accuracy while keeping the total storage capacity essentially unchanged. Meanwhile, by recording the position index information of the first data element through the index memory, the multiply-accumulate calculation circuit can dynamically determine the precision format corresponding to the current data based on the index information after reading the data, and then adaptively execute the multiply-accumulate operation that matches the precision format. This allows high-precision data elements and low-precision data elements to complete their respective operations in the same set of hardware circuits without setting up independent computing cores. This reduces hardware resource overhead while ensuring the accuracy of the operation and improves the processing efficiency of the in-memory computing architecture for mixed precision data.

[0009] In addition, the location index information includes a flag bit and address information. The flag bit indicates whether the corresponding storage array contains the first data element, and the address information indicates the position of the first data element in the storage array. The multiply-accumulate calculation circuit includes a decoder connected to the index memory, which receives the flag bit and address information and generates a mode signal. The mode signal indicates the precision format corresponding to the data currently read from the storage array. Thus, by splitting the location index information into two dimensions, flag bit and address information, the decoder can quickly determine whether a high-precision data element exists in the current storage block and its specific location with extremely low hardware complexity. The generated mode signal can directly drive the subsequent calculation circuit to switch the precision format, avoiding the need for element-by-element scanning or complex decoding of the entire storage block, and reducing the latency overhead of mode determination.

[0010] Furthermore, the multiply-accumulate calculation circuit includes: a lookup table, a multiplexer connected to the lookup table, the lookup table storing multiple candidate operation results determined based on data read from the storage array; the multiple candidate operation results include at least the operation result obtained by adding at least two components of the data read from the storage array; the multiplexer selects the corresponding operation result from the multiple candidate operation results in the lookup table based on the received activation value and outputs it. Thus, by pre-storing multiple candidate operation results, including component addition results, in the lookup table, the arithmetic operation that originally needed to be performed in real time by multipliers and adders in the multiply-accumulate operation is transformed into a lookup selection operation based on activation values. The multiplexer directly selects the output from the candidate results based on the activation value, eliminating the large array of full adders required for real-time multiplication operations. This significantly reduces the dynamic power consumption and circuit area of ​​the operation unit while achieving equivalent multiply-accumulate functionality.

[0011] In addition, the in-memory computing circuit also includes an exponent processing unit, which includes an exponent comparison module, an exponent subtraction module, and an exponent addition module connected to the exponent comparison module and the exponent subtraction module. The exponent addition module receives the exponent portion of the data read from the memory array and the exponent portion of the activation value, sums them to obtain multiple exponent sums, and outputs the multiple exponent sums to the exponent comparison module and the exponent subtraction module. The exponent comparison module receives multiple exponent sums, determines the maximum exponent sum, and outputs the maximum exponent sum to the exponent subtraction module. The exponent subtraction module receives multiple exponent sums and the maximum exponent sum, calculates the difference between the maximum exponent sum and each exponent sum, generates a shift alignment signal, and outputs it. The exponent comparison module and the exponent subtraction module share a logic unit, and the output of the shared logic unit participates in both the determination process of the maximum exponent sum in the exponent comparison module and the calculation process of the difference in the exponent subtraction module. Therefore, by sharing logic units between the exponent comparison module and the exponent subtraction module, the operation results of the same logic unit can be used for both the selection of the maximum exponent sum and the calculation of the exponent difference. This avoids setting up independent logic circuits in the two functional modules to perform operations with the same operational nature. Without sacrificing calculation accuracy, the number of redundant logic gates in the exponent processing unit is reduced, effectively reducing circuit area and power consumption. At the same time, the delay of exponent processing is shortened by reducing the number of logic levels on the critical path. Attached Figure Description

[0012] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0013] Figure 1 This is a block diagram of the overall architecture of a mixed-precision in-memory computing circuit according to an embodiment of the present invention in fixed-point precision format; Figure 2 This is a comparative schematic diagram of a mixed-precision data storage method and an existing mass production scheme in a storage array according to an embodiment of the present invention; Figure 3 This is a block diagram of the overall architecture of a mixed-precision in-memory computing circuit in floating-point precision format according to an embodiment of the present invention; Figure 4 This is a detailed structural schematic diagram of an in-memory computing array according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the mixed-precision decoding process and mode configuration according to an embodiment of the present invention; Figure 6 This is a schematic diagram of the data format when the first precision format is BF16 according to an embodiment of the present invention; Figure 7 This is a schematic diagram of the configuration of a lookup table-based multiply-accumulate circuit under different precision modes according to an embodiment of the present invention. Figure 8 This is a schematic diagram of the processing flow and shared logic unit mechanism of an index processing unit according to an embodiment of the present invention; Figure 9 This is a schematic diagram of the gate-level design and non-maximum exponent masking process of a bit serial exponent comparator based on a shared XOR gate according to an embodiment of the present invention. Figure 10 This is a schematic diagram of the circuit-level design of an exponential subtractor based on a shared XOR gate according to an embodiment of the present invention; Figure 11 This is a flowchart of a data processing method according to an embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details are presented in the embodiments of the present invention to facilitate a better understanding of the invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various variations and modifications based on the following embodiments. The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.

[0015] In the description of the embodiments of the present invention, the terms "first," "second," etc., are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly defined.

[0016] In the description of the embodiments of this invention, the term "and / or" is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A exists, A and B exist simultaneously, and B exists. In addition, the character " / " in this document generally indicates that the related objects before and after it have an "or" relationship.

[0017] In the description of the embodiments of the present invention, the so-called "electrical connection" or "electrical connection" refers to a connection between structures for current or signal transmission when the device is in operation, since the connecting material is a conductive material.

[0018] In the description of this embodiment of the invention, "data element" refers to the operand participating in the multiplication-accumulation operation in the in-memory circuit. As an example, the data element can be the weight parameter data in a neural network model (e.g., a large language model). Correspondingly, "activation value" refers to the input data that is multiplied and accumulated with the weight parameters, which is usually obtained by processing the output of the previous layer through an activation function during the neural network inference process.

[0019] To facilitate understanding of the variable symbols used in this invention specification and accompanying drawings, this document uniformly adopts a compound naming rule of 'main variable + subscript + bit index', specifically as follows: (1) Main variable, represented in a regular font, is used to indicate the type of physical quantity represented by the symbol. For example, M represents mantissa, W represents weight, IN represents input, and E represents exponent.

[0020] (2) Subscripts, represented in subscript format, follow the main variable and are used to further define the specific instance or semantic attribute of the physical quantity. Subscripts can be numbers (e.g., 1, 2, 3 in W1, W2, W3, representing the 1st, 2nd, and 3rd weight variables in the storage location, respectively), or letters or letter combinations (e.g., M). IN In the IN clause, it indicates that the mantissa is an active value; E W E IN In the index, W and IN represent the weight and activation value, respectively; E MAX E SUM E SHIFT In this context, MAX, SUM, and SHIFT represent the maximum exponential sum, exponential sum, and shift-aligned signal, respectively.

[0021] (3) Bit index, represented by inline square brackets '[]' immediately following the variable (not subscript format), is used to indicate the specific bit position or bit segment within the variable. For example, W1[3] represents the 3rd bit within the first weight variable W1; E MAX [i] represents the maximum exponent and E. MAX The i-th bit inside.

[0022] All variable symbols involved in this invention specification and drawings follow the above naming rules; the meanings of subscripts and position indices of each specific symbol will not be repeated in subsequent embodiments and drawings.

[0023] One embodiment of the present invention relates to a mixed-precision in-memory computing circuit, which can be integrated into a chip. This chip can be applied to artificial intelligence inference acceleration hardware platforms, including but not limited to edge computing devices, data center accelerator cards, or embedded smart terminals for large language model inference. From a system-level perspective, this chip, as the core computing unit of the inference acceleration hardware, integrates one or more compute-in-memory macros (CIMs). Each CIM contains multiple compute-in-memory arrays, and each CIM array further includes functional modules such as a storage subarray, a multiply-accumulate computation array, an index storage and decoding module, an exponent processing unit, a product shifter, a channel-level adder tree, and a normalization unit. In practical deployment scenarios, large language models need to perform large-scale matrix multiply-accumulate operations during the inference phase. These operations involve multiplying and accumulating weight data stored in memory with externally input activation values. In the compute-in-memory architecture, weight data is pre-written into the storage array, activation values ​​are input externally, and multiply-accumulate operations are performed near the storage units, thereby avoiding the latency and energy consumption caused by repeatedly moving data between the processor and memory.

[0024] With in-memory computing technology now widely accepted in the industry, a well-known macro-level limitation faced by those skilled in the art is that when the model's weight data, after quantization, contains data elements of varying precision, existing in-memory computing architectures exhibit a trade-off between storage efficiency and computational precision when handling such mixed-precision data. Furthermore, they suffer from high hardware overhead and computational latency during mixed-precision computations. These issues have been widely observed within the industry, prompting practitioners to explore improvements in multiple directions.

[0025] However, when faced with the aforementioned macroscopic defects, those skilled in the art usually seek solutions along two conventional lines of thought, but both of these paths have fundamental limitations.

[0026] The first conventional approach is to use an outlier-aware quantization algorithm. Outliers in the weight matrix that have a critical impact on inference accuracy are retained in a high-precision format (e.g., BF16, or Bfloat16, 16 bits wide), while other normal values ​​are quantized in a low-precision format (e.g., INT4, 4 bits wide), and the two types of data are stored separately. Those skilled in the art generally believe that as long as outliers can be accurately identified and retained, a mixed-precision quantization scheme can achieve a balance between accuracy and efficiency. However, this understanding overlooks a hidden structural contradiction: when outliers and normal values ​​are stored independently using different bit-width data formats, outliers require additional coordinate indices (e.g., row numbers, column numbers) and bitmask information to identify their position in the weight matrix. This index information itself occupies additional storage space, offsetting the storage savings offered by low-precision quantization. More importantly, because the number of outliers in each data block is dynamically changing (blocks without outliers only contain normal INT4 values, while blocks with outliers have additional BF16 data and index information inserted), this variable-length storage format fundamentally disrupts the continuity and regularity of storage access. The core design premise of in-memory computing architecture is precisely that storage access must maintain a fixed width and fixed address step to achieve efficient burst access and parallel reads. The variable-length format prevents the memory controller from predicting the data boundary of the next read with a fixed step size, forcing the introduction of dynamic address calculation and data rearrangement logic. This not only reduces memory bandwidth utilization but also causes the data path between the storage side and the computing side to lose its deterministic timing relationship. This is a structural defect for in-memory computing arrays that aim for single-cycle or fixed-latency computation. In other words, the conventional approach of separate storage has a fundamental conflict with the design constraints of in-memory computing architecture in terms of the regularity of the storage format. This conflict is often overlooked in conventional quantization algorithm research because algorithm researchers usually assume that storage access is flexible and variable, and do not pay attention to the rigid constraints of the underlying hardware on the access mode.

[0027] The second conventional approach is to support mixed-precision computation at the in-memory macro level through mode switching or a dual-core architecture: either by allowing a single CIM macro to switch between all-integer (INT) and all-floating-point (FP) modes, or by setting up independent INT and FP computation cores to handle data of corresponding precisions respectively. Those skilled in the art generally believe that mixed-precision computation can be achieved simply by equipping data of different precisions with corresponding computational units. However, this understanding overlooks another problem: in actual weight matrices, the distribution of outliers is element-wise random, meaning outliers can appear anywhere in the matrix, rather than clustering in certain rows or columns. Macro-level mode switching selects the mode at the granularity of the entire CIM macro, which is completely inadequate to handle this element-wise random distribution; while a dual-core architecture can handle data of two different precisions separately, it requires sorting the data by precision and distributing it to different cores before computation. This sorting process itself requires additional decoding circuitry and data transport paths, and because outliers typically account for a low percentage in the matrix, the workload between the two cores is severely unbalanced. The FP core remains idle most of the time, resulting in wasted hardware resources.

[0028] Based on the above analysis, the applicant identified that the underlying cause of the aforementioned macro-level defects is not a simple trade-off between accuracy and efficiency, but rather stems from three interconnected micro-level contradictions: First, there is a contradiction between the storage addressing method for outliers and the rigid requirement for access regularity in in-memory compute arrays. Existing solutions either sacrifice storage regularity to accommodate variable-length formats or sacrifice accuracy to maintain a uniform format; the two are mutually exclusive. Second, the mismatch between the element-level random distribution characteristics of outliers and the granularity of macro-level or core-level mode switching results in an operational granularity much larger than the data granularity, making it impossible to achieve adaptive precision at the element level. Third, in the exponent alignment operation during the accumulation of mixed-precision product terms, there is a significant overlap in operations between the comparison logic and the subtraction logic (both require bit-by-bit comparison of the exponent value). However, existing solutions implement these two as independent modules, leading to the redundant deployment of logic gates and unnecessary overhead in terms of both area and latency.

[0029] The negative impacts of the three micro-level contradictions mentioned above on energy efficiency and power consumption are relatively prominent. For example, in high-energy-efficiency, low-power application scenarios (such as edge computing devices, wearable smart terminals, and battery-powered embedded inference platforms), the chip's power budget is extremely limited. Every bit of redundant computation and every redundant logic gate will directly translate into unacceptable energy consumption. The memory access fragmentation caused by the first contradiction requires the memory controller to perform additional dynamic address calculations and data rearrangement operations. These operations themselves do not produce effective computational output, but consume valuable energy budget, reducing the effective computational throughput per unit of energy consumption (i.e., energy efficiency). The large amount of idle high-precision computing cores in the dual-core architecture caused by the second contradiction means that the static leakage power of the core continues to exist throughout the inference process but is not effectively utilized, resulting in a significant decrease in the overall energy efficiency of the chip. The redundant deployment of exponential comparison logic and subtraction logic caused by the third contradiction not only increases the circuit area (thus increasing static power consumption), but also increases dynamic power consumption due to the switching activity of redundant logic gates. Furthermore, the extended critical path forces the circuit to operate at a lower clock frequency, further restricting the computational performance per unit of power consumption. Therefore, to achieve hybrid precision in-memory computing in high-energy-efficiency, low-power scenarios, it is not only necessary to solve the three micro-contradictions mentioned above, but also to consistently adhere to the principles of minimizing power consumption and maximizing energy efficiency in the design of the solution, so that every circuit design optimization can be transformed into quantifiable power reduction and energy efficiency improvement.

[0030] To address the aforementioned problems, this invention proposes a mixed-precision in-memory computing circuit. Its core design concept is as follows: At the storage level, the storage locations of the storage array are uniformly divided according to the bit width of the low-precision format. This allows high-precision data elements to be naturally embedded into the storage structure with low-precision format as the basic granularity, covering the storage locations of adjacent low-sensitivity low-precision data elements. This achieves the coexistence of mixed-precision data while maintaining storage access regularity. At the indexing level, a compact index memory records the position information of high-precision data elements, enabling the multiply-accumulate calculation circuit to dynamically determine the precision format of the data at the element level with extremely low decoding overhead. At the computation level, the multiply-accumulate calculation circuit adaptively executes operations matching the precision format based on the index information, allowing data elements of different precisions to complete their respective operations within the same hardware circuit.

[0031] The specific implementation details of the embodiments of the present invention are described below. The following content is only for the convenience of understanding the implementation details and is not necessary for implementing this solution.

[0032] Example 1 like Figure 1As shown, the mixed-precision in-memory computing circuit provided in this embodiment of the invention includes: a storage array, an index memory, and a multiply-accumulate computing circuit electrically connected to the storage array and the index memory.

[0033] The storage array is used to store data elements to be processed. The storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format, wherein the bit width of the first precision format is greater than the bit width of the second precision format.

[0034] It should be noted that the specific types of the first and second precision formats are not limited; they can be any combination of floating-point, integer, or other data representation formats. As an example, the first precision format can be BF16 floating-point format, with a bit width of 16 bits, including 1 sign bit, 8 exponent bits, and 7 mantissa bits; the second precision format can be INT4 integer format, with a bit width of 4 bits. As another example, the first precision format can also be FP16 (half-precision floating-point) or INT8 (8-bit integer) format, and the second precision format can also be FP4 (4-bit floating-point) or INT2 (2-bit integer) format, as long as the bit width of the first precision format is greater than that of the second precision format. As yet another example, the first precision format can be INT8 integer format, and the second precision format can be INT4 integer format; in this case, both are integer formats, not a combination of a floating-point and an integer. Therefore, this invention does not impose mandatory limitations on the data type (floating-point or integer) of the first and second precision formats; the core difference lies in their bit width.

[0035] Storage locations in the storage array are divided according to the bit width of the second-precision format. That is, the smallest storage granularity in the storage array corresponds to the bit width of the second-precision format. The first data element, because its bit width is larger than the bit width of the second-precision format, occupies multiple consecutive storage locations.

[0036] like Figure 2As shown, the example uses BF16 (16 bits) for the first precision format and INT4 (4 bits) for the second precision format. Each storage location in the memory array is 4 bits wide (i.e., the bit width of INT4). A first data element in BF16 format occupies four consecutive storage locations (16 ÷ 4 = 4). When a set of data in the memory array contains an outlier (e.g., the value 73.5), this outlier is stored in BF16 format, occupying four consecutive storage locations; however, these four storage locations could originally store four second data elements in INT4 format. In other words, the first data element covers and occupies the storage locations corresponding to adjacent, less sensitive second data elements. Since the second data elements originally corresponding to the adjacent storage locations covered by the first data element contribute relatively little to the overall calculation result, exchanging the storage space of these less sensitive data elements for high-precision preservation of highly sensitive data elements effectively improves the overall calculation precision while keeping the total storage capacity essentially unchanged.

[0037] The read bit width of the storage array is an integer multiple of the bit width of the first-precision format. This setting ensures that each read from the storage array contains exactly an integer number of first-precision format data elements, or multiple corresponding second-precision format data elements, thus guaranteeing the regularity of storage access. Continuing with the BF16 / INT4 example, if the read bit width is set to 16 bits (i.e., 1 times the BF16 bit width), each 16-bit read will either contain one first data element in BF16 format or four second data elements in INT4 format. If the read bit width is set to 32 bits (i.e., 2 times the BF16 bit width), each 32-bit read can contain two first data elements in BF16 format, eight second data elements in INT4 format, or a mixture of both.

[0038] In embodiments of the present invention, the storage array may include one or more groups, each group comprising multiple rows and columns of storage units. As an example, the storage array may include N groups (N is an integer greater than or equal to 1), each group being 16×16 bits in size, i.e., each group contains 16 rows, each row having a storage width of 16 bits. Within each group, a mixture of first and second data elements can be flexibly stored, or all first data elements can be stored, or all second data elements can be stored; the storage method is unrestricted. For example, in a 16×16-bit group, the 16 bits of a row can be stored entirely as one first data element in BF16 format, or entirely as four second data elements in INT4 format, or different precision storage methods can be used in different rows within the same group.

[0039] The above storage method allows high-precision data elements to be naturally embedded in a storage structure with low-precision format as the basic granularity. Without introducing additional coordinate indexes or variable-length storage formats, it maintains the regularity and continuity of storage access and avoids the memory access fragmentation problem caused by the bit width difference of mixed precision data.

[0040] In the storage array, when the storage array includes a first data element stored in a first-precision format and a second data element stored in a second-precision format (i.e., the "and" case), the storage array simultaneously contains data elements of both precisions; when the storage array includes only the first data element stored in a first-precision format (i.e., only the first data element), all data in the storage array is stored in a high-precision format; when the storage array includes only the second data element stored in a second-precision format (i.e., only the second data element), all data in the storage array is stored in a low-precision format. All three cases fall under the data storage modes supported by the storage array of this invention, and the multiply-accumulate calculation circuit can perform corresponding operations on data in any of these modes.

[0041] In embodiments of the present invention, the storage medium of the storage array can be Static Random-Access Memory (SRAM), Dynamic Random-Access Memory (DRAM), flash memory, magnetoresistive random-access memory, or other storage media suitable for in-memory computing architectures. As an example, the storage array is constructed based on SRAM bit cells, and each SRAM bit cell can adopt a 6-transistor (6T) structure, an 8-transistor (8T) structure, or other bit cell topologies suitable for in-memory computing. Figure 4 As shown, the memory array may further include word line drivers and bit line drivers connected to each memory cell, used to drive word lines and bit lines during the write phase to write data to a specified memory location, and to drive word lines during the computation phase to read the stored data; and readout circuitry and readout bit line prechargers, used to precharge the readout bit lines during the readout phase and sense the electrical signals output by the memory cells, converting them into digital logic levels for use by subsequent multiply-accumulate computation circuits. The aforementioned word line drivers, bit line drivers, readout circuitry, and prechargers are peripheral circuits required for the memory array to implement read, write, and computation functions. Those skilled in the art can select appropriate circuit topologies based on the specific memory medium type and process node, and will not be elaborated further here.

[0042] The index memory stores the location index information of the first data element in the storage array. This location index information is used to indicate the specific location of the first data element in the storage array. When data is written to the storage array, it can be determined which locations store the first data element, and the corresponding location index information is written to the index memory.

[0043] By recording the position index information of the first data element in the index memory, the multiply-accumulate calculation circuit can dynamically determine the precision format corresponding to the current data based on the index information after reading the data, without having to perform element-by-element scanning or complex decoding of the entire storage block, thereby reducing the latency overhead of mode determination.

[0044] The multiply-accumulate calculation circuit is used to determine the precision format corresponding to the data read from the storage array based on the position index information, and to perform multiply-accumulate operations corresponding to the determined precision format, and output the calculation result.

[0045] In one embodiment of the present invention, the in-memory computing circuit may further include a sparse detector connected to the read end of the memory array, used to detect data elements with a value of zero in the data read from the memory array. When the sparse detector detects that a data element is zero, it can generate a gating signal to skip the multiply-accumulate operation corresponding to that data element, thereby avoiding performing invalid multiplication and addition operations on zero-value data and reducing dynamic power consumption during the operation. In the weight data of large language models, after quantization processing, there is often a certain proportion of zero-value elements (especially after adopting a storage strategy of covering adjacent low-sensitivity data elements with outliers, the original low-precision data element corresponding to the covered position is zero). The sparse detector can make full use of this data sparsity characteristic to further improve the computational efficiency.

[0046] In embodiments of the present invention, the multiply-accumulate calculation circuit can be implemented in various ways. As an example, the multiply-accumulate calculation circuit can employ a shared set of computing hardware, which adapts to different precision operations through mode switching (e.g., when high-precision and low-precision formats have reusable operational logic, such as INT8 and INT4 sharing the same shift-add logic). As another example, the multiply-accumulate calculation circuit can also set up two sets of computing hardware corresponding to the first and second precision formats respectively. After determining the precision format of the current data based on the position index information, the data is distributed to the corresponding computing hardware for execution. Both implementation methods can achieve the technical effect of adaptively performing multiply-accumulate operations according to the precision format.

[0047] Furthermore, the in-memory computing circuit of this invention achieves optimizations in energy efficiency and power consumption. In the multiply-accumulate operation stage, by using an operation structure based on lookup tables and multiplexers, real-time multiplication operations are transformed into lookup table selection operations, eliminating the switching activities of a large number of full adders in traditional multiplier arrays, and significantly reducing the dynamic power consumption of multiply-accumulate operations. In the exponentiation processing stage, by sharing logic units between the exponent comparison module and the exponent subtraction module, the number of redundant logic gates is reduced, which not only reduces the static power consumption corresponding to the circuit area, but also reduces the dynamic power consumption generated by the switching activities of logic gates, while shortening the critical path delay, enabling the circuit to operate at a higher operating frequency and improving the computational throughput per unit power consumption. At the architectural level, by adaptively processing data elements of different precisions using the same set of hardware circuits, the area waste and static power consumption overhead caused by setting up independent computing cores are avoided, thus fully improving the utilization of hardware resources. The above-mentioned multi-dimensional power consumption optimizations make the in-memory computing circuit of this invention particularly suitable for application scenarios such as edge computing devices and embedded smart terminals with strict power budget constraints, achieving high-energy-efficiency acceleration of large-scale language model inference while ensuring the accuracy of mixed-precision operations.

[0048] It should be noted that the in-memory computing circuit of this invention supports floating-point and fixed-point formats (fixed-point formats include integer and fixed-point decimal formats). When both the first and second precision formats are fixed-point formats (e.g., the first precision format is INT8 and the second precision format is INT4), since the data elements in the fixed-point format do not contain an exponent, no exponent alignment operation is required between the product terms. Therefore, the in-memory computing circuit does not need to be configured with an exponent processing unit and a product shifter. Figure 3 As shown, in the fixed-point precision format configuration, the mantissa operation result output by the multiplexer in the multiply-accumulate calculation circuit is directly transmitted to the channel-level adder tree for step-by-step accumulation. The accumulation result of the channel-level adder tree is then processed by the normalization unit before outputting the final calculation result. The data path under this configuration is simpler than that of the floating-point precision format, omitting the hardware overhead of the exponent addition module, exponent comparison module, exponent subtraction module, and product shifter, thereby further reducing circuit area and power consumption in fixed-point arithmetic scenarios.

[0049] When the first precision format and / or the second precision format are floating-point formats (e.g., the first precision format is BF16 floating-point format and the second precision format is INT4 integer format), because the data elements in the floating-point format include an exponent part, the exponents of different product terms may be different, requiring exponent alignment of each product term before accumulation. For example... Figure 3As shown, in the floating-point precision format configuration, the in-memory circuit also includes an exponent processing unit and a product shifter. The exponent processing unit receives the exponent portion of the data read from the memory array and the exponent portion of the activation value input externally, and generates a shift alignment signal. The product shifter performs a shift alignment operation on the mantissa operation results of each product term output by the multiply-accumulate calculation circuit according to the shift alignment signal. After shift alignment, each product term is then transmitted to the channel-level adder tree for step-by-step accumulation, and finally the normalization unit outputs the final calculation result.

[0050] As can be seen, the memory-based circuit of this invention can flexibly configure the data path according to the precision format adopted: in fixed-point precision format, a simplified data path without exponential processing (such as...) is used. Figure 1 As shown); in floating-point precision format, a complete data path including an exponent processing unit and a product shifter is used (e.g. Figure 3 (As shown). The multiply-accumulate calculation circuits (including lookup tables and multiplexers) in both configurations can be reused. The only difference is whether an exponent alignment stage is required. This allows the same memory circuit architecture to adapt to the application requirements of both fixed-point and floating-point operations.

[0051] like Figure 4 As shown in the embodiments of the present invention, when the data processed by the in-memory computing circuit involves floating-point precision format, the in-memory computing integrated array in the in-memory computing circuit also includes an exponent processing unit, a product shifter, a channel-level adder tree, and a normalization unit connected to the multiply-accumulate calculation circuit, to form a complete floating-point calculation data path. When the data processed by the in-memory computing circuit is all in fixed-point precision format, the in-memory computing circuit includes a channel-level adder tree and a normalization unit connected to the multiply-accumulate calculation circuit, and the output of the multiply-accumulate calculation circuit is directly transmitted to the channel-level adder tree.

[0052] The product shifter, connected to the output of the multiply-accumulate circuit, receives the mantissa calculation results of each product term from the multiply-accumulate circuit and the shift alignment signal from the exponent processing unit (if present). It then performs a shift alignment operation on the mantissa calculation results of each product term based on the shift alignment signal. This shift alignment aligns the mantissas of product terms with different exponents to a unified exponent base, providing correct operands for subsequent accumulation operations. As an example, in a storage array containing multiple groups (such as 32 CIM subarrays), the product shifter performs shift alignment on each of the 32 groups of product terms separately.

[0053] A channel-level adder tree, connected to the output of a product shifter, receives the shifted and aligned product terms and performs a step-by-step accumulation operation, outputting the accumulated result. The hierarchical structure of the adder tree can be configured according to the number of product terms involved in the accumulation. As an example, for 32 shifted and aligned product terms, the adder tree can use a 5-level binary tree structure (log₂32=5) to add each term pairwise, ultimately outputting a single accumulated result. The specific implementation of the adder tree is not limited to a binary tree structure; it can also use Wallace trees, Dada trees, or other adder topologies suitable for large-scale parallel accumulation.

[0054] The normalization unit, connected to the output of the channel-level addition tree, normalizes the accumulated result and outputs the final calculation result. Normalization may include leading zero detection, mantissa shifting, and exponent adjustment to ensure the final output conforms to the target data format. As an example, the normalization unit may include a leading one detector and a normalization shifter. The leading one detector determines the position of the most significant bit in the accumulated result, and the normalization shifter shifts the mantissa left or right based on this position, adjusting the exponent accordingly, and outputting the normalized floating-point result.

[0055] In one embodiment of the present invention, the entire critical computation path, starting from reading data from the storage array, passing through the multiply-accumulate calculation circuit, the product shifter, the channel-level adder tree, and finally outputting the result in the normalization unit, is implemented using pure combinational logic without using pipelined registers. Since the entire data path completes all operations from data reading to final result output within a single clock cycle, seamless integration of exponentiation and mantissa processing is ensured, avoiding the additional latency and potential pipeline stalls introduced by pipelined registers, thereby maximizing computational throughput while ensuring computational correctness. In another embodiment of the present invention, the aforementioned critical computation path can also incorporate one or more stages of pipelined registers to adapt to higher clock frequency requirements; those skilled in the art can choose according to specific timing constraints and throughput requirements.

[0056] Therefore, high-precision data elements and low-precision data elements can complete their respective operations in the same in-memory computing circuit without setting up completely independent computing cores. This reduces hardware resource overhead while ensuring computational accuracy and improves the processing efficiency of the in-memory computing architecture for mixed-precision data.

[0057] Example 2 In another embodiment of the present invention, the difference from embodiment 1 is that the specific operation method of the multiply-accumulate calculation circuit under different precision formats is further defined.

[0058] When the location index information indicates that the data read from the storage array corresponds to the first precision format, the multiply-accumulate calculation circuit takes the read data as at least one first data element and performs a multiplication operation with the corresponding activation value.

[0059] It should be noted that the number of first data elements in the data read from the storage array depends on the ratio between the read bit width of the storage array and the bit width of the first precision format. As an example, if the read bit width of the storage array is 16 bits and the first precision format is BF16 (16 bits), then the data read at one time corresponds to exactly one first data element, which is multiplied by its corresponding activation value. As another example, if the read bit width of the storage array is 32 bits and the first precision format is BF16 (16 bits), then the data read at one time can correspond to two first data elements, which are then multiplied by their respective activation values.

[0060] When the location index information indicates that the data read from the storage array corresponds to the second-precision format, the multiply-accumulate circuit treats the read data as at least two second data elements and performs multiply-accumulate operations with the corresponding external input activation values. Continuing with the BF16 / INT4 example above, if the read bit width is 16 bits, the read 16 bits of data are parsed into four INT4 format second data elements, which are then multiplied-accumulate operations with the four corresponding INT4 format activation values.

[0061] The above method allows the same multiply-accumulate calculation circuit to flexibly switch the operation mode according to the precision format of the current data: for high-precision data, it performs multiplication operation with a single activation value, and for low-precision data, it performs multiply-accumulate operation with multiple activation values, thus balancing calculation precision and efficiency.

[0062] Example 3 In another embodiment of the present invention, the difference from embodiment 2 is that the specific structure of the location index information and the decoder in the multiply-accumulate calculation circuit are further defined.

[0063] like Figure 5 As shown, the location index information includes a flag bit and address information. The flag bit indicates whether the corresponding storage array contains the first data element. The address information indicates the location of the first data element in the storage array.

[0064] As an example, consider a data block containing 32 storage locations. The location index information of this data block can include a 1-bit flag (OFlag) and 5-bit address information (OAdd). A flag of 1 indicates that the data block contains the first data element, while a flag of 0 indicates that the data block does not contain the first data element. The 5-bit address information is used to indicate the starting position of the first data element within the 32 storage locations (with a value ranging from 0 to 31).

[0065] The multiply-accumulate calculation circuit includes: a decoder connected to the index memory, which receives flag bits and address information and generates a mode signal that indicates the precision format of the data currently read from the memory array.

[0066] As an example, the decoder can be a binary-on-one-hot decoder. Taking the high-order bits of the address information as an example, which generate a mode signal through the binary-on-one-hot decoder, this mode signal directly indicates whether the subarray containing the currently read data should operate in first-precision format or second-precision format. When the flag bit is 0 or the mode signal indicates that the current subarray does not contain the first data element, the read data is parsed and processed in second-precision format; when the flag bit is 1 and the mode signal indicates that the current subarray contains the first data element, the read data is parsed and processed in first-precision format.

[0067] By splitting the location index information into two dimensions, flag bits and address information, the decoder can quickly determine whether there are high-precision data elements and their specific locations in the current memory block with extremely low hardware complexity. The generated mode signal can directly drive the subsequent calculation circuit to switch the precision format, avoiding the need to perform element-by-element scanning or complex decoding of the entire memory block.

[0068] Example 4 In another embodiment of the present invention, the difference from embodiment 3 is that the activation selector and format conversion module in the multiply-accumulate calculation circuit are further defined.

[0069] like Figure 5 As shown, the multiply-accumulate calculation circuit also includes: an activation selector connected to the index memory, and a format conversion module connected to the activation selector.

[0070] The activation selector receives address information and multiple externally input activation values, selects the target activation value corresponding to the first data element from the multiple activation values ​​according to the address information, and outputs the target activation value to the format conversion module.

[0071] As an example, in a scenario where the first precision format is BF16 and the second precision format is INT4, when the mode signal indicates that the current operation mode is the first precision format, the 16 bits of data read are used as a weight in the BF16 format. At this time, among the inputs that originally corresponded to the four INT4 activation values, the activation selector selects one INT4 activation value corresponding to the first data element as the target activation value based on the low-order bits of the address information (such as OAdd[1:0]).

[0072] The format conversion module converts the data format of the target activation value and outputs it in a format that matches the first precision format.

[0073] It should be noted that the data format of the activation value does not necessarily have to be exactly the same as the precision format of the weight data (i.e., the first data element). The format of the activation value is determined by the format requirements of the multiply-accumulate calculation circuit that processes it. In other words, the multiply-accumulate calculation circuit that processes its activation value can have the same or different requirements for the input format of the weight data and the activation value, as long as the multiply-accumulate calculation circuit can correctly perform the calculation.

[0074] As an example, continuing with the scenario where the first precision format is BF16 and the second precision format is INT4, when the multiply-accumulate calculation circuit requires the input activation value to be in BF16 format, the format conversion module converts the selected INT4 activation value into an equivalent value in BF16 format, ensuring that the converted value is numerically equivalent to the original INT4 value, thereby enabling the converted activation value to correctly perform multiplication with the weights in BF16 format.

[0075] As another example, when the design of the multiply-accumulate circuit allows for receiving activation value inputs in a different format than the weight data (e.g., the multiply-accumulate circuit has the internal capability to perform operations on operands of different bit widths), the format conversion module can convert the target activation value into a specific format required by the multiply-accumulate circuit, which may differ from the first precision format. For example, if the first data element is a weight in BF16 format, and the multiply-accumulate circuit requires the input activation value to be in INT4 format, the format conversion module will convert the target activation value to INT4 format for output, and the multiply-accumulate circuit will internally perform the multiplication operation between the BF16 format weight and the INT4 format activation value. This configuration avoids the unnecessary bit width expansion introduced by extending low-precision activation values ​​to high-precision formats. Within the limits of the hardware logic of the multiply-accumulate circuit, the operation is completed with a more compact data bit width, thereby reducing the conversion overhead of the format conversion module and the bit width requirements of the intermediate operations of the multiply-accumulate circuit, further reducing dynamic power consumption.

[0076] By combining the activation selector and the format conversion module, in the first precision format operation mode, the multiply-accumulate calculation circuit can accurately select the target activation value corresponding to the first data element from multiple activation values, and convert the target activation value into the corresponding format according to the format requirements of the multiply-accumulate calculation circuit for the input activation value, thus avoiding calculation errors caused by format mismatch. At the same time, there is no need to maintain a separate activation value input path for high precision calculation, which simplifies the hardware design.

[0077] In one embodiment of the present invention, when the first precision format is a floating-point format conforming to the IEEE-754 standard (such as BF16), the mantissa portion of the first data element adopts an implicit leading value encoding method, that is, the highest bit of the mantissa is implicitly "1" and is not explicitly stored in the storage array. Figure 6 As shown, taking the BF16 format as an example, a BF16 data element contains 1 sign bit S, 8 exponent bits E, and 7 mantissa bits Man, and its actual value can be represented as v = (-1). S ×2 E-127 ×(1.Man), where "1.Man" represents the combination of the implicit leading "1" of the mantissa and the 7-bit explicit mantissa. During the multiply-accumulate operation, the implicit leading "1" needs to be restored and concatenated with the explicit mantissa to form the complete 8-bit mantissa value M. W [7:0]={1,Man W [6:0]}. As an example, the complete 8-bit mantissa can be further broken down into the high 4 bits M. WH [3:0] and the lower 4 bits M WL [3:0] is used to adapt to the input format of a lookup table-based multiply-accumulate circuit. Similarly, the mantissa of the activation value after format conversion also undergoes the same implicit leading value recovery and splitting operation to form M. IN [7:0]={1,Man IN [6:0]} and split into M INH [3:0] and M INL [3:0].

[0078] It should be noted that the above-described encoding method for implicit leading values ​​applies to normalized floating-point numbers in the IEEE-754 standard. For subnormalized floating-point numbers, the implicit leading value of the mantissa is "0" instead of "1". In the current embodiment of the present invention, as an example, the processing of subnormalized floating-point numbers may not be supported temporarily to simplify circuit design complexity. In another embodiment of the present invention, the multiply-accumulate circuit can also be extended to support subnormalized floating-point numbers by adding logic to determine whether the exponent part is all zero. When the exponent is all zero, the implicit leading value is set to "0" to achieve floating-point operations that fully comply with the IEEE-754 standard. Those skilled in the art can choose according to actual precision requirements.

[0079] Example 5 In another embodiment of the present invention, a specific implementation of the multiply-accumulate calculation circuit is further defined, namely, an operational structure based on a lookup table and a multiplexer.

[0080] It should be noted that the lookup table-based multiply-accumulate circuit described in Embodiment 5 is a specific circuit implementation scheme proposed in this invention that can reduce the number of full adders and lower operational power consumption. In other implementations, conventional multipliers and adders can be used to directly perform real-time multiply-accumulate operations on the data read from the memory array and the activation value, without using a lookup table. The technical advantage of the lookup table-based approach described in this embodiment is that it transforms real-time multiplication operations into selection operations based on lookup tables, thereby eliminating the large array of full adders required for real-time multiplication operations. While achieving equivalent multiply-accumulate functionality, it significantly reduces the dynamic power consumption and circuit area of ​​the arithmetic unit.

[0081] The multiply-accumulate calculation circuit includes: a lookup table and a multiplexer connected to the lookup table.

[0082] A lookup table stores multiple candidate operation results determined based on data read from the storage array. These candidate operation results include at least one operation result obtained by adding at least two components of the data read from the storage array.

[0083] like Figure 7 As shown, as an example, in the second-precision format (INT4) operation mode, the data read from the storage array contains multiple INT4 weights (such as W0 and W2). The lookup table pre-stores the following candidate results: W0 (corresponding to activation value bit combination 10), W2 (corresponding to activation value bit combination 01), W0+W2 (corresponding to activation value bit combination 11), and 0 (corresponding to activation value bit combination 00). Here, W0+W2 is the result of adding at least two components (W0 and W2) from the read data. This addition operation can be performed using a pre-computed adder.

[0084] The pre-computation adder is connected to a lookup table to pre-calculate the addition result between multiple weight components based on the weight data read from the storage array before the operation, and uses this addition result as a candidate entry in the lookup table. As an example, in the second precision format (INT4) operation mode, the pre-computation adder receives two weight values, W0 and W2, calculates W0 + W2, and stores the result in the corresponding entry in the lookup table; in the first precision format (BF16) operation mode, the pre-computation adder receives the lower half of the mantissa, M... WL and 2×M WL (Obtained through shifting), calculate M WL +2×MWL =3×M WL The result is then stored in the corresponding entry in the lookup table. The pre-computed adder can be activated via a multiplexer, meaning that addition is only performed when the corresponding candidate entry needs to be used, thus avoiding unnecessary dynamic power consumption overhead.

[0085] It should be noted that the generation method of candidate operation results in the lookup table is not limited to the implementation of the pre-computation adder described above. In other embodiments, all candidate results can be pre-calculated by software and directly written into the storage unit of the lookup table, or other hardware logic (such as a combination of shift registers and adders) can be used to generate candidate results. All of the above different candidate result generation methods can achieve the technical effect of transforming real-time multiplication operations into lookup table selection operations.

[0086] As another example, in the first precision format (BF16) operation mode, the read data is a weight in BF16 format, with its mantissa M... W [7:0] can be split into the high half M WH [3:0] and the lower half of M WL [3:0]. The lookup table pre-stores the following candidate results: M WL (Corresponding to the activation value bit combination 01), 2×M WL (Generated through shift operations, corresponding to the active value bit combination 10), 3×M WL (Generated via a pre-computed adder, corresponding to activation value bit combination 11), and 0 (corresponding to activation value bit combination 00). Similarly, M WH The corresponding candidate results are configured in the same way.

[0087] A multiplexer selects and outputs the corresponding operation result from multiple candidate operation results in a lookup table based on the received activation value. In other words, the multiplexer uses the activation value (or a portion of the activation value) as a selection signal to directly select the output from the pre-stored candidate results in the lookup table, without performing real-time multiplication operations.

[0088] The multiply-accumulate circuit described above, based on lookup tables and multiplexers, transforms the arithmetic operations that originally required multipliers and adders to perform in real time into lookup-based selection operations based on activation values, thus eliminating the need for a large array of full adders required for real-time multiplication operations.

[0089] In one embodiment of the invention, the configuration of the selection signal of the multiplexer varies depending on the current precision format operation mode.

[0090] In the operation mode corresponding to the first precision format, the multiplexer uses the two adjacent bits of the activation value as selection signals to select from multiple candidate operation results. For example... Figure 7 As shown in the middle floating-point mode, taking BF16 operation mode as an example, the mantissa M of the activation value. IN [7:0] is divided into multiple combinations of two adjacent pairs, i.e., M IN [2i+1] and M IN [2i] (i=0 to 3), each multiplexer receives a set of two adjacent bits as selection signals. Since these two adjacent bits are from different bit segments of the same activation value, in this mode, each multiplexer essentially selects candidate results based on the local bit information within the same activation value. This corresponds to the operational semantics of multiplying a single weight by a single activation value in the first precision format—that is, the candidate calculation results of the same weight mantissa are selected and aggregated segment by segment from the bit segments of the same activation value.

[0091] In the operation mode corresponding to the second precision format, the multiplexer uses one bit of each of the two activation values ​​as a selection signal to select from multiple candidate operation results. For example... Figure 7 As shown in the middle integer mode, taking the INT4 operation mode as an example, the data read from the storage array contains multiple INT4 weights (such as W0, W1, W2, W3), with corresponding activation values ​​IN0, IN1, IN2, and IN3, respectively. In the lookup table, W0 and W2 are paired to generate candidate results (i.e., 0, W0, W2, W0+W2). Each multiplexer receives one bit of IN0[i] and one bit of IN2[i] (i.e., taking one bit from each of the two different activation values) as selection signals. Since these two bits come from the same position of two different activation values, in this mode, each multiplexer essentially selects candidate results jointly based on the corresponding bits of the two different activation values. This corresponds to the operational semantics of multiplying multiple weights by multiple activation values ​​and then summing them in the second-precision format. That is, by associating the candidate combination of two weights with the corresponding bits of the two activation values, the calculation of two product terms is completed simultaneously in a single lookup operation.

[0092] The difference between the two selection signal configurations essentially stems from the different organization of the computational objects in the two precision modes: in the first precision format mode, a single high-precision weight is multiplied by a single activation value, so the selection signal is taken from adjacent bits of the same activation value; in the second precision format mode, multiple low-precision weights are multiplied and accumulated by multiple activation values, so the selection signal is taken from corresponding bits of different activation values. Through this selection signal adaptation strategy based on the computation mode, the same lookup table and multiplexer hardware can be reused in both precision modes without needing to set up separate computation paths for different precisions, further reducing circuit area and hardware cost.

[0093] Example 6 In another embodiment of the present invention, the difference from embodiment 5 is that the parallel structure of the multiplexer is further defined.

[0094] The multiply-accumulate circuit includes multiple multiplexers. Each multiplexer receives the corresponding bit segment of the activation value as its own selection signal, selects the corresponding operation result from multiple candidate operation results in parallel, and outputs them respectively.

[0095] As an example, taking an 8-bit activation value, this 8-bit activation value can be divided into four 2-bit segments. Four multiplexers each receive one 2-bit segment as a selection signal. During operation, the four multiplexers can simultaneously use their respective bit segments to select the output from the candidate results of the lookup table in parallel, thus completing the entire lookup selection operation within a single clock cycle.

[0096] Compared to performing table lookup and selection operations bit by bit serially, the parallel architecture enables the multiplexer to complete the selection operations corresponding to multiple bit segments simultaneously within a single clock cycle, significantly improving the throughput and computational efficiency of multiply-accumulate operations.

[0097] Example 7 In another embodiment of the invention, an exponent processing unit in the memory circuit is further defined.

[0098] like Figure 8 As shown, the memory circuit also includes an exponent processing unit, which includes an exponent comparison module, an exponent subtraction module, and an exponent addition module connected to the exponent comparison module and the exponent subtraction module.

[0099] The exponent addition module receives the exponent portion of the data read from the storage array and the exponent portion of the activation value, sums them to obtain multiple exponent sums, and outputs the multiple exponent sums to the exponent comparison module and the exponent subtraction module.

[0100] As an example, in BF16 format, each weight has an 8-bit exponent part E. W Each activation value has an 8-bit exponent part E. IN The exponential addition module will add each pair of E W and E IN Adding them together gives the exponent and E. SUM When there are 32 pairs of weights and activation values, the exponential addition module outputs 32 E's. SUM value.

[0101] It should be noted that the exponent processing unit uses a product alignment strategy for floating-point multiplication and accumulation operations. The product alignment strategy refers to using the maximum value among all exponent sums (i.e., the maximum exponent sum E). MAXUsing the maximum exponent sum as the alignment benchmark, the mantissa results of each product term are right-shifted and aligned according to the difference between their exponent sum and the maximum exponent sum, ensuring that each product term has a unified exponent benchmark before entering the addition tree for accumulation. Compared to the pairwise alignment strategy (i.e., aligning only two product terms each time before adding), the product alignment strategy aligns all product terms at once using the maximum exponent sum as the unified benchmark, achieving accumulation with minimal data truncation, thus achieving higher computational accuracy under the same hardware resources.

[0102] The exponent comparison module receives multiple exponent sums, determines the largest exponent sum, and outputs it to the exponent subtraction module. The purpose of determining the largest exponent sum is that in floating-point multiplication and accumulation operations, the exponents of different product terms may be different. The largest exponent sum is used as an alignment benchmark to align the mantissas of each product term before accumulation, achieving accurate accumulation with minimal data truncation.

[0103] The exponent subtraction module receives multiple exponent sums and a maximum exponent sum, calculates the difference between the maximum exponent sum and each exponent sum, generates a shift alignment signal, and outputs it. This shift alignment signal indicates the number of bits the mantissa of each product term needs to be shifted to the right, aligning the mantissa of each product term to the reference corresponding to the maximum exponent sum, thereby ensuring the correctness of subsequent accumulation operations.

[0104] The exponent comparison module and the exponent subtraction module share a common logic unit (e.g., Figure 8 The shared XOR gate in the shared logic unit is used to determine the maximum sum of exponents in the exponent comparison module and to calculate the difference in the exponent subtraction module.

[0105] By sharing a logic unit between the exponent comparison module and the exponent subtraction module, the operation result of the same logic unit is used for both the selection of the maximum exponent sum and the calculation of the exponent difference. This avoids setting up independent logic circuits in the two functional modules to perform operations with the same operational nature. Without sacrificing calculation accuracy, the number of redundant logic gates in the exponent processing unit is reduced, effectively reducing circuit area and power consumption. At the same time, the delay of exponent processing is shortened by reducing the number of logic levels on the critical path.

[0106] It should be noted that the implementation of shared logic units is not limited to a specific type of logic gate. It can be an XOR gate, a NOR gate, an AND gate, or other logic units that can simultaneously perform comparison and subtraction operations.

[0107] Example 8 In another embodiment of the present invention, the difference from embodiment 7 is that the specific implementation method of the shared logic unit is further defined.

[0108] like Figure 9 and Figure 10 As shown, the shared logic unit is an XOR gate.

[0109] In the exponent comparison module, the XOR gate performs an XOR operation on the corresponding bits of each exponent sum and the corresponding bits of the candidate maximum exponent sum. The result of the operation is used as a bypass arbitration signal. When the current bit of a certain exponent sum is different from the candidate maximum exponent sum, the subsequent bits of the certain exponent sum are blocked from participating in the comparison through the gating signal.

[0110] As an example, such as Figure 9 As shown, the exponent comparison module uses a bit-sequential comparison strategy to determine the maximum exponent sum. During the comparison process, it proceeds bit by bit, starting from the most significant bit: at each bit, a NOR gate is used to determine the value E of the candidate maximum exponent sum at that bit. MAX [i]; Subsequently, the XOR gate will each E SUM The corresponding bit and E MAX [i] Perform an XOR operation; if a certain E SUM The position and E MAX If [i] are different, the XOR operation output is 1, which serves as a bypass arbitration signal to enable this E. SUM The bypass is achieved by shielding the E through an AND gate. SUM The subsequent bits are pre-set to logic "0" before participating in the NOR operation, so that they are no longer included in the comparison of subsequent bits. This process proceeds from the most significant bit to the least significant bit, and the final unmasked E is retained. SUM That is, the maximum exponent sum.

[0111] In the exponential subtraction module, the result of the XOR gate is used as the propagation signal in the carry-bypass adder to participate in the calculation of the difference between the maximum exponential sum and each exponential sum.

[0112] like Figure 10 As shown, the exponential subtraction module adopts a carry-bypass adder architecture, using a standard propagation signal (P) and a generated signal (G) to perform subtraction operations. The propagation signal P is defined as E... MAX [i]⊕( The XOR operation in this module shares the same XOR gate as the XOR operation in the exponent comparison module. That is, the output of the XOR gate serves both as the bypass arbitration signal in the exponent comparison module and as the propagation signal for the carry-bypass adder in the exponent subtraction module. The generated signal G is defined as E. MAX [i]&( E is completed by cascading a full adder constructed from P and G signals. MAX [8:0]+( The calculation of E, that is, the realization of E MAX -E SUM The subtraction operation ultimately outputs the shift alignment signal E. SHIFT .

[0113] Example 9 Another embodiment of the present invention relates to a data processing method applied to the mixed-precision in-memory circuit described above. Figure 11 As shown, the method includes the following steps.

[0114] In step S1, the data is stored in a storage array. The storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format, wherein the bit width of the first precision format is greater than the bit width of the second precision format; storage locations in the storage array are divided according to the bit width of the second precision format; the first data element occupies multiple consecutive storage locations; and the read bit width of the storage array is an integer multiple of the bit width of the first precision format.

[0115] In step S2, the position index information of the first data element in the storage array is stored in the index memory.

[0116] In step S3, data is read from the storage array and the corresponding position index information is read from the index memory. The precision format corresponding to the read data is determined according to the position index information. The read data and the activation value are subjected to a multiplication and accumulation operation corresponding to the determined precision format, and the operation result is output.

[0117] The steps described above are for clarity only. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the protection scope of this invention. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, without changing the core design of the algorithm and process, are also within the protection scope of this invention.

[0118] It is not difficult to see that this embodiment is a method embodiment corresponding to the circuit embodiment described above, and this embodiment can be implemented in conjunction with the circuit embodiment described above. The relevant technical details mentioned in the circuit embodiment described above are still valid in this embodiment, and will not be repeated here to reduce repetition. Correspondingly, the relevant technical details mentioned in this embodiment can also be applied to the circuit embodiment described above.

[0119] Example 10 Another embodiment of the present invention relates to a chip including the mixed-precision in-memory computing circuit described above. This chip can be applied to scenarios such as accelerating large-scale language model inference, deep neural network inference, and edge intelligent computing. By integrating the aforementioned mixed-precision in-memory computing circuit, efficient in-memory computing processing of mixed-precision data is achieved at the chip level.

[0120] Furthermore, the examples mentioned in the above embodiments can be freely combined, and any combination can be understood as an embodiment. The terms "embodiment" or "example" appearing in various locations in the specification do not necessarily refer to the same embodiment, nor are they independent or alternative embodiments mutually exclusive with other embodiments. Those skilled in the art will understand that the embodiments described herein can be combined with other embodiments.

[0121] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.

Claims

1. A hybrid-precision-based compute-in-memory circuit, comprising: include: A storage array, an index memory, and a multiply-accumulate calculation circuit electrically connected to the storage array and the index memory; The storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format; wherein the bit width of the first precision format is greater than the bit width of the second precision format. The storage locations in the storage array are divided according to the bit width of the second precision format; the first data element occupies multiple consecutive storage locations; the read bit width of the storage array is an integer multiple of the bit width of the first precision format; The index memory stores the position index information of the first data element in the storage array; The multiply-accumulate calculation circuit is used to determine the precision format corresponding to the data read from the storage array based on the position index information, and to perform a multiply-accumulate operation corresponding to the determined precision format, and output the calculation result. Specifically, when the location index information indicates that the data read from the storage array corresponds to the first precision format, the multiply-accumulate calculation circuit treats the read data as at least one of the first data elements and performs a multiplication operation with a corresponding activation value; when the location index information indicates that the data read from the storage array corresponds to the second precision format, the multiply-accumulate calculation circuit treats the read data as at least two of the second data elements and performs a multiply-accumulate operation with multiple corresponding externally input activation values ​​respectively.

2. The hybrid-precision-based compute-in-memory circuit of claim 1, wherein, The location index information includes a flag bit and address information. The flag bit indicates whether the corresponding storage array contains the first data element, and the address information indicates the position of the first data element in the storage array. The multiply-accumulate calculation circuit includes: The decoder connected to the index memory receives the flag bit and the address information and generates a mode signal, which indicates the precision format of the data currently read from the storage array.

3. The hybrid-precision-based compute-in-memory circuit of claim 2, wherein, The multiply-accumulate calculation circuit further includes: an activation selector connected to the index memory, and a format conversion module connected to the activation selector; The activation selector receives the address information and multiple externally input activation values, selects a target activation value corresponding to the first data element from the multiple activation values ​​according to the address information, and outputs the target activation value to the format conversion module. The format conversion module converts the data format of the target activation value and outputs it in a format that matches the first precision format.

4. The mixed-precision based compute-in-memory circuit according to any one of claims 1 to 3, wherein, The multiply-accumulate calculation circuit includes: a lookup table and a multiplexer connected to the lookup table. The lookup table stores multiple candidate operation results determined based on data read from the storage array; the multiple candidate operation results include at least the operation result obtained by adding at least two components of the data read from the storage array; The multiplexer, based on the received activation value, selects and outputs the corresponding operation result from the multiple candidate operation results in the lookup table.

5. The hybrid-precision-based compute-in-memory circuit of claim 4, wherein, The multiply-accumulate calculation circuit includes multiple multiplexers; The multiple multiplexers each receive the corresponding bit segment of the activation value as their respective selection signal, and select the corresponding operation result from the multiple candidate operation results in parallel and output them respectively.

6. The mixed-precision based compute-in-memory circuit according to any one of claims 1 to 3, wherein, The in-memory circuit further includes an exponent processing unit, which includes an exponent comparison module, an exponent subtraction module, and an exponent addition module connected to the exponent comparison module and the exponent subtraction module. The exponential addition module receives the exponential part of the data read from the storage array and the exponential part of the activation value, sums them to obtain multiple exponential sums, and outputs the multiple exponential sums to the exponential comparison module and the exponential subtraction module. The exponent comparison module receives the plurality of exponent sums, determines the maximum exponent sum from the plurality of exponent sums, and outputs the maximum exponent sum to the exponent subtraction module; The exponent subtraction module receives the sum of the plurality of exponents and the maximum sum of exponents, calculates the difference between the maximum sum of exponents and each of the sums of exponents, generates a shift alignment signal and outputs it. The exponent comparison module and the exponent subtraction module share a logic unit, and the output of the shared logic unit participates in both the determination of the maximum exponent sum in the exponent comparison module and the calculation of the difference in the exponent subtraction module.

7. The hybrid-precision-based compute-in-memory circuit of claim 6, wherein, The shared logic unit is an XOR gate; In the exponent comparison module, the XOR gate performs an XOR operation on the corresponding bits of each of the exponent sums and the corresponding bits of the candidate maximum exponent sum. The result of the operation is used as a bypass arbitration signal. When the current bit of a certain exponent sum is different from the candidate maximum exponent sum, the subsequent bits of the certain exponent sum are blocked from participating in the comparison by the gating signal. In the exponential subtraction module, the result of the XOR gate is used as a propagation signal in the carry-bypass adder to participate in the calculation of the difference between the maximum exponential sum and each of the exponential sums.

8. A data processing method applied to the hybrid-precision-based computing and storage circuit according to any one of claims 1 to 7, characterized in that, include: Data is stored in a storage array; wherein the storage array includes a first data element stored in a first precision format and / or a second data element stored in a second precision format, the bit width of the first precision format being greater than the bit width of the second precision format; the storage locations in the storage array are divided according to the bit width of the second precision format; the first data element occupies multiple consecutive storage locations; the read bit width of the storage array is an integer multiple of the bit width of the first precision format; Store the position index information of the first data element in the storage array into the index memory; Data is read from the storage array, and the corresponding position index information is read from the index memory. The precision format corresponding to the read data is determined according to the position index information. The read data and the activation value are subjected to a multiplication-accumulation operation corresponding to the determined precision format, and the operation result is output. Specifically, when the location index information indicates that the data read from the storage array corresponds to the first precision format, the read data is used as at least one first data element and multiplied with a corresponding activation value; when the location index information indicates that the data read from the storage array corresponds to the second precision format, the read data is used as at least two second data elements and multiplied and accumulated with multiple corresponding externally input activation values ​​respectively.

9. A chip, characterized by Includes a mixed-precision memory circuit as described in any one of claims 1 to 7.

Images