Mixed-precision floating-point digital storage macro and multi-chip module

Abstract

The invention discloses a mixed precision floating point digital memory macro and multi-chip module. The module comprises an HFP-CIM array, an input activation memory, a maximum index finder, an adder tree, an FP combiner and a C2C connector, the HFP-CIM array is respectively connected with the input activation memory, the maximum index finder and the adder tree; the FP combiner is respectively connected with the maximum index finder and the adder tree; the C2C connector is connected with the input activation memory, and is connected with the HFP-CIM array and the FP combiner through the multiplexer. The HFP-CIM array with segmented read word lines is designed, the floating point precision and EMR of the HFP-CIM array are reconfigurable, and the throughput of chips in various HFP formats can be improved. Meanwhile, the reconfigurable in-memory accumulator can parallelly calculate indexes and mantissas so as to reduce delay of FP operation and further improve throughput. According to the invention, the off-chip storage access of a single chip is minimized, the multi-chip module based on core particles is supported to have two-dimensional extensible heterogeneous data streams, data reuse among chips is enhanced, and the off-chip storage access of the multi-chip module is inhibited.

Description

Technical Field

[0001] The present invention relates to the field of in-memory computing technology, and in particular to a mixed-precision floating-point digital storage computing macro and a multi-chip module. Background Art

[0002] As AI models become more complex and large, such as large language models (such as GPT, BERT, etc.) and diffusion models, their requirements for numerical accuracy are getting higher and higher. These models usually need to use floating point (FP) format to ensure sufficient accuracy to support complex mathematical operations and gradient update processes.

[0003] Recent CIM (Compute-in-Memory) designs have focused on supporting classic high-precision floating-point formats, such as single-precision floating point (FP32), half-precision floating point (FP16), and brain floating point (BF16). Although these formats provide high numerical accuracy, they also bring significant storage overhead and bit operation burden, thus limiting the room for performance improvement.

[0004] In addition, when processing different layers and tensor shapes in Convolutional Neural Networks (CNN) and Transformer architecture networks, traditional CIM macro units need to frequently access external memory (External Memory Access). This repeated access not only increases latency, but also consumes a lot of bandwidth resources. Especially in a multi-chip module (Multi-Chip Module) environment, external memory access and inter-chip communication bandwidth become bottlenecks that restrict system scalability. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a mixed-precision floating-point digital storage and calculation macro and a multi-chip module.

[0006] A first aspect of the present invention provides a mixed-precision floating-point digital storage and calculation macro, comprising an HFP-CIM array, an input activation memory, a maximum index finder, an addition tree, an FP combiner and a C2C connector; the HFP-CIM array is respectively connected to the input activation memory, the maximum index finder and the addition tree; the FP combiner is respectively connected to the maximum index finder and the addition tree; the C2C connector is connected to the input activation memory, and connects the HFP-CIM array and the FP combiner through a multiplexer.

[0007] Further, the C2C connector includes an input activation receiver, an input activation transmitter, a partial and output activation receiver, a partial and output activation transmitter, a weight receiver and a weight transmitter; wherein the input activation receiver and the input activation transmitter are connected to the input activation memory; the weight receiver is connected to the HFP-CIM array through a multiplexer; the weight transmitter is connected to the HFP-CIM array through a multiplexer; the partial and output activation transmitter is connected to the partial and output activation receiver, the HFP-CIM array and the FP combiner through a multiplexer;

[0008] The input activation receiver is used to write the external input activation stream into the input activation memory;

[0009] The weight receiver is used to write the external weight stream into the HFP-CIM array;

[0010] The partial and output activation receiver is used to transmit an external output activation stream to the partial and output activation transmitter;

[0011] The partial and output activation transmitter is used to output the output activation stream provided by the HFP-CIM array, the FP combiner and the partial and output activation receiver;

[0012] The input activation transmitter is used to output the input activation stream provided by the input activation memory;

[0013] The weight transmitter is used to output the weight stream provided by the HFP-CIM array.

[0014] Furthermore, the input activation memory includes a plurality of static random access memories for storing input activation streams; the static random access memories transmit the input activation streams to the HFP-CIM array via a multiplexer.

[0015] Further, the HFP-CIM array includes a plurality of levels, each of which includes a plurality of SRAM rows, a RWL decoder, a WWL decoder, a WBL driver, a RBL driver, a reconfigurable in-memory accumulator, and a mantissa alignment module;

[0016] In each level, the RWL decoder, the WWL decoder, the WBL driver and the RBL driver are each connected to each SRAM row; the RWL decoder is also connected to the input activation memory; the RBL driver is also connected to the reconfigurable in-memory accumulator, the partial and output activation transmitter and the weight transmitter through a multiplexer; the reconfigurable in-memory accumulator is connected to the maximum exponent finder and the mantissa alignment module; the mantissa alignment module is connected to the maximum exponent finder and the addition tree;

[0017] The RWL decoder is used to receive the input activation stream of the input activation memory and transmit the input activation stream to each SRAM row for storage;

[0018] The WBL driver is used to selectively write the bit lines in the SRAM row;

[0019] The WWL decoder is used to select a word line in the SRAM row for selective writing;

[0020] The SRAM row is used to store weight data or partial sum data or output activation data according to the writing results of the WBL driver and the WWL decoder and perform in-memory calculation operations to obtain output signals to be output to the RBL driver;

[0021] The RBL driver is used to receive the output signal of the SRAM row, and transmit the output signal to the reconfigurable in-memory accumulator through the multiplexer; the RBL driver is also used to generate an output data stream, and transmit the output data stream to the partial and output activation transmitter through the multiplexer; the RBL driver is also used to transmit the weight data stream to the weight transmitter through the multiplexer;

[0022] The reconfigurable in-memory accumulator is used to receive the input activation stream transmitted by the input activation memory and the output signal of the HFP-CIM array, perform parallel exponential addition operation and input activation bit-level accumulation operation, obtain an exponential addition signal and an accumulation signal, and transmit the exponential addition signal to the maximum exponent finder, and transmit the accumulation signal to the mantissa alignment module;

[0023] The mantissa alignment module is used to perform a mantissa alignment operation on the accumulated signal output by the reconfigurable in-memory accumulator, and obtain an aligned mantissa signal which is transmitted to an addition tree for channel-level accumulation operation.

[0024] Furthermore, the RWL decoder includes a read enable configuration or an input activation configuration; each of the SRAM rows includes a plurality of static random access memory cells, each of which is divided into an exponent sector and a mantissa sector; the static random access memory cells of the exponent sector perform exponential addition operations, and the static random access memory cells of the mantissa sector perform mantissa multiplication operations.

[0025] Furthermore, the read word lines of the static random access memory cells in the SRAM row are connected via switches for exponent-mantissa ratio; and the SRAM rows are connected via switches for floating-point precision;

[0026] In the HFP-CIM array, the ratio of the SRAM row addition sector and the accumulation sector static random access memory unit can be changed by switching control of the exponent-mantissa ratio to adjust the exponent-mantissa ratio of the HFP-CIM array; the SRAM row can be arbitrarily reorganized by switching the floating-point precision to reconstruct the granularity of the HFP-CIM array, thereby supporting arbitrary floating-point precision.

[0027] Furthermore, the reconfigurable in-memory accumulator includes a plurality of full adders and a group of ping-pong SRAMs; the full adders are arranged in a row and connected to the ping-pong SRAMs via a multiplexer;

[0028] The full adder and ping-pong SRAM are divided into an addition sector and an accumulation sector according to the setting of the HFP-CIM array; the full adders of the addition sector and the accumulation sector, and the ping-pong SRAMs of the addition sector and the accumulation sector are connected through a switch for the exponent-mantissa ratio; the full adder of the addition sector performs an exponential addition operation, obtains an exponential addition signal, and outputs it to the maximum exponent finder to search for the maximum exponent; the ping-pong SRAM of the accumulation sector cyclically performs read and write operations and cooperates with the full adder of the accumulation sector to perform an accumulation operation of the mantissa at the input activation bit level, and obtains an accumulation signal, which is output to the mantissa module.

[0029] Further, the full adder of the addition sector receives the output signal from the RBL driver transmission SRAM row and the exponential signal of the input data stream, performs exponential addition operation, and outputs the exponential addition signal to the exponential sector of the ping-pong SRAM;

[0030] The full adder of the accumulation sector receives the output signal from the RBL driver transmission SRAM row and the mantissa accumulation partial sum signal stored in one row of the accumulation sector ping-pong SRAM in the previous cycle, performs mantissa accumulation operation, and outputs the accumulated partial sum to another row of the ping-pong SRAM.

[0031] The second aspect of the present invention discloses a multi-chip module based on a core particle, the multi-chip module has the macro, the multi-chip module has a weight fixed in-memory calculation mode and an output fixed in-memory calculation mode;

[0032] Among them, the weight fixed in-memory calculation mode specifically includes:

[0033] Setting the static random access memory cells in the SRAM row to store weight data, and setting the external memory access calculation priority that minimizes the weight;

[0034] According to the kernel parameter and channel parameter settings, the weight data is loaded into the static random access memory cells of the SRAM row;

[0035] The input activation stream is vertically transmitted to each HFP-CIM macro through the input activation memory; in-memory calculation is performed in each HFP-CIM array, the in-memory calculation result of the previous HFP-CIM array is used as the input of the next HFP-CIM array, and the in-memory calculation result of the last HFP-CIM array is transmitted to the RBL driver as an output signal;

[0036] Output fixed memory calculation modes include:

[0037] Setting the static random access memory cells in the SRAM row to store weight data or partial data and / or output activation data, and setting an external memory access calculation priority that minimizes output activation data;

[0038] Setting multiple horizontal HFP-CIM array groups, each HFP-CIM array group performs in-memory calculations in parallel;

[0039] The weight data is horizontally transmitted in each HFP-CIM macro, and the input activation stream is vertically transmitted to each HFP-CIM macro; in-memory calculation is performed in each horizontal HFP-CIM array group, and the in-memory calculation result of the last HFP-CIM array group is transmitted to the RBL driver as an output signal.

[0040] Furthermore, the external memory access calculation priority for minimizing weights and / or the external memory access calculation priority for minimizing output activation data are determined based on the external artificial intelligence model structure of the external input data stream source.

[0041] The embodiments of the present invention have the following beneficial effects: The present invention utilizes the concept of low-precision hybrid floating point (HFP) to design an HFP-CIM array with a segmented reading word line (Segmented Reading Word Line), whose floating point precision and EMR (Exponent-to-Mantissa Ratio) are reconfigurable, which can improve the data throughput of chips under various HFP formats. At the same time, the reconfigurable in-memory accumulator (RIMA) can calculate the exponent and mantissa in parallel (Parallel-EM) to reduce the delay of FP operations and further improve data throughput. The present invention minimizes single-chip off-chip storage access and supports a multi-chip module based on a core particle with a two-dimensional scalable heterogeneous data flow (Scalable Heterogeneous Data flows, SHD), enhances data reuse between chips and suppresses off-chip storage access of multi-chip modules.

[0042] Additional aspects and advantages of the present invention will be given in part in the following description, and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 It is a schematic diagram of the macro architecture of the present invention.

[0045] Figure 2 It is a schematic diagram of the SRAM row structure based on SRWL in the HFP-CIM array of the present invention.

[0046] Figure 3 It is a schematic diagram of the working structure of the SRAM row in the HFP-CIM array of the present invention.

[0047] Figure 4 It is a schematic diagram of the structure of the reconfigurable in-memory accumulator of the present invention.

[0048] Figure 5 It is a schematic diagram of the process of realizing parallel calculation of exponent and mantissa of the present invention.

[0049] Figure 6 It is a schematic diagram of the working architecture of the multi-chip module of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0051] In-memory computing is a processing architecture that integrates storage units and computing units. It can process data directly at the data storage location, significantly improve computing efficiency, and is widely used in the calculation of artificial intelligence networks such as CNN and Transformer.

[0052] To improve computational efficiency, Figure 1 As shown, the first embodiment of the present invention provides a mixed precision floating point digital storage and calculation macro, including an HFP-CIM array, an input activation memory, a maximum index finder, an addition tree, an FP combiner, and a C2C (Chip-to-Chip) connector.

[0053] The HFP-CIM array is connected to the input activation memory, the maximum index finder and the addition tree respectively; the FP combiner is connected to the maximum index finder and the addition tree respectively; the C2C connector is connected to the input activation memory, and connects the HFP-CIM array and the FP combiner through a multiplexer.

[0054] The present invention provides support for efficient HFP calculation formats (such as HFP4 / 6 / 8 / 16) in CIM calculations through a reconfigurable HFP-CIM array, and improves the data throughput of a chip through parallel calculation of exponents and mantissas.

[0055] The following specifically describes each module of the present invention:

[0056] C2C connector: The C2C connector of the present invention is used for data exchange between the internal macro architecture and the external artificial intelligence network. The external data includes input activation data and weight data, which are used for in-memory calculation and weight distribution respectively. The C2C connector of the present invention includes an input activation receiver (Input Activate Receiver, IA-RX), an input activation transmitter (Input Activate Transmitter, IA-TX), a partial and output activation receiver (Partial-sum / OutputActivate Receiver, P-sum / OA-RX), a partial and output activation transmitter (Partial-sum / OutputActivate Transmitter, P-sum / OA-TX), a weight receiver (Weight Activate Receiver, W-RX) and a weight transmitter (Weight Activate Transmitter, W-TX).

[0057] The input activation receiver and the input activation transmitter are connected to the input activation memory; the weight receiver is connected to the HFP-CIM array through a multiplexer; the weight transmitter is connected to the HFP-CIM array through a multiplexer; and the partial and output activation transmitters are connected to the partial and output activation receivers, the HFP-CIM array and the FP combiner through a multiplexer.

[0058] Specifically, the input activation receiver of the embodiment of the present invention is used to write an external input activation stream into the input activation memory; the weight receiver is used to write an external weight stream into the HFP-CIM array; the partial and output activation receiver is used to transmit the external output activation stream to the partial and output activation transmitter; the partial and output activation transmitter is used to output the output activation stream provided by the HFP-CIM array, the FP combiner and the partial and output activation receiver; the input activation transmitter is used to output the input activation stream provided by the input activation memory; the weight transmitter is used to output the weight stream provided by the HFP-CIM array.

[0059] Input activation memory: In the embodiment of the present invention, the input activation memory is used to store input activation data. The input activation memory includes multiple static random access memories for storing input activation streams; the static random access memories transmit the input activation streams to the HFP-CIM array through a multiplexer.

[0060] HFP-CIM array: In the embodiment of the present invention, a 4×8-channel HFP-CIM array is provided, corresponding to 8 channels of input activation memory, maximum index finder and addition tree.

[0061] Each HFP-CIM array includes multiple levels, each of which includes 16 SRAM rows (Static Random-Access Memory), RWL decoder (Reading Word Line Decoder), WWL decoder (Wrtiing Word Line Decoder), WBL driver (Writing Bit Line Driver), RBL driver (Reading Bit Line Driver), reconfigurable in-memory accumulator and mantissa alignment module.

[0062] In each level, the RWL decoder, WWL decoder, WBL driver and RBL driver are connected to each SRAM row respectively; the RWL decoder is also connected to the input activation memory; the RBL driver is also connected to the reconfigurable in-memory accumulator, the partial and output activation transmitter and the weight transmitter through the multiplexer; the reconfigurable in-memory accumulator is connected to the maximum exponent finder and the mantissa alignment module; the mantissa alignment module is connected to the maximum exponent finder and the addition tree;

[0063] The RWL decoder is used to receive the input activation stream from the input activation memory and transmit the input activation stream to each SRAM row for storage;

[0064] The WBL driver is used to selectively write to the bit lines in an SRAM row;

[0065] The WWL decoder is used to select the word line in the SRAM row for selective writing;

[0066] The SRAM row is used to store weight data or partial sum data or output activation data according to the writing results of the WBL driver and the WWL decoder and perform in-memory calculation operations to obtain output signals to be output to the RBL driver;

[0067] The RBL driver is used to receive the output signal of the SRAM row and transmit the output signal to the reconfigurable in-memory accumulator through the multiplexer; the RBL driver is also used to generate an output data stream and transmit the output data stream to the partial and output activation transmitter through the multiplexer; the RBL driver is also used to transmit the weight data stream to the weight transmitter through the multiplexer;

[0068] The reconfigurable in-memory accumulator is used for receiving the input activation stream transmitted by the input activation memory and the output signal of the HFP-CIM array, performing parallel exponential addition operation and input activation bit-level accumulation operation, obtaining an exponential addition signal and an accumulation signal, and transmitting the exponential addition signal to the maximum exponent finder, and transmitting the accumulation signal to the mantissa alignment module;

[0069] The mantissa alignment module is used to perform mantissa alignment operation on the accumulated signal output by the reconfigurable in-memory accumulator, and the obtained aligned mantissa signal is transmitted to the addition tree for channel-level accumulation operation.

[0070] In an embodiment of the present invention, the RWL decoder includes a read enable configuration or an input activation (IA) configuration. Figure 2 , 3 As shown, each SRAM row includes a plurality of static random access memory cells (SRAM cells, Figure 2 The SRAM cells in the exponential sector perform exponential addition operations, and the SRAM cells in the accumulative sector perform mantissa multiplication operations. Among them, the SRAM cells configured for read enable by the RWL decoder are divided into the E sector, and the SRAM cells configured for IA are divided into the M sector.

[0071] Specifically, the read word line (RWL) of the static random access memory cell in the SRAM row is switched by switching the exponent-mantissa ratio (S EMR ) connection; SRAM rows are connected by a floating point precision switch (S Grain ) connection. The RWL decoder operates by S EMR The ratio of the SRAM cells in the E and M sectors of the SRAM row can be changed to adjust the exponent-mantissa ratio of the HFP-CIM array; Grain The SRAM rows can be arbitrarily reorganized to reconstruct the granularity of the HFP-CIM array to support arbitrary floating point precision.

[0072] For example, when the external data is in FP16 format, the embodiment of the present invention can adjust S EMR and S GrainThe SRAM row is adjusted to 16 units per row, wherein the first unit is used for allocation of the read word line, the following 5 units are divided into the E sector, and the 10 units are divided into the M sector, so as to realize the reconstruction of the HFP-CIM array. When the external data is in the FP8 format, the embodiment of the present invention can adjust the SRAM row to 8 units per row by adjusting SEMR and SGrain, wherein the first unit is used for allocation of the read word line, the following 4 units are divided into the E sector, and the 3 units are divided into the M sector, so as to realize the reconstruction of the HFP-CIM array.

[0073] Reconfigurable in-memory accumulator: The RIMA of the present invention includes multiple full adders and a group of ping-pong SRAMs. Figure 4 As shown, the full adders are arranged in a row and connected to the ping-pong SRAM through a multiplexer. The ping-pong SRAM in the RIMA of the present invention is an 8T-SRAM with a ping-pong structure. That is, the data is alternately written into two different SRAM modules (referred to as A and B). When a group of data is being written into SRAM A, the data that has been previously written into SRAM B can be read out and processed at the same time. Once the writing of SRAM A is completed, the input data stream will switch to SRAM B for new data writing, and the data in SRAM A will begin to be read out and processed. In this way, by continuously switching the read and write operations between the two SRAMs, seamless data processing is achieved, pauses in the data processing process are avoided, and the throughput and efficiency of the system are improved.

[0074] The full adder and ping-pong SRAM are divided into an addition sector and an accumulation sector according to the setting of the HFP-CIM array; the full adders of the addition sector and the accumulation sector, and the ping-pong SRAMs of the addition sector and the accumulation sector are connected through a switch for the exponent-mantissa ratio; the full adder of the addition sector performs an exponential addition operation, and the exponential addition signal is output to the maximum exponent finder for searching the maximum exponent; the ping-pong SRAM of the accumulation sector performs read and write operations cyclically and cooperates with the full adder of the accumulation sector to perform an accumulation operation of the mantissa at the input activation bit level, and the accumulated signal is output to the mantissa module.

[0075] As a specific embodiment, the full adder of the addition sector receives the output signal from the RBL driver transmission SRAM row and the exponential signal of the input data stream, performs exponential addition operation, and then outputs the exponential addition signal to the exponential sector of the ping-pong SRAM.

[0076] The full adder of the accumulation sector receives the output signal from the RBL driver transmission SRAM row and the mantissa accumulation partial sum signal stored in a row of the accumulation sector ping-pong SRAM in the previous cycle, and after performing the mantissa accumulation operation, outputs the accumulated partial sum to another row of the ping-pong SRAM.

[0077] like Figure 5As shown, the RIMA workflow of the embodiment of the present invention is as follows:

[0078] S1: The addition sector of RIMA performs exponential addition, and at the same time, the accumulation sector of RIMA accumulates the mantissa multiplication results of the HFP-CIM array, that is, performs feature level accumulation;

[0079] S2: The result of the exponential addition is transmitted to the maximum exponent comparator (digital comparator tree) to search for the maximum exponent;

[0080] S3: The result of the mantissa accumulation is aligned based on the exponent data;

[0081] S4: The aligned mantissa is fed to the adder tree, which performs channel-level accumulation.

[0082] Existing floating-point multiplication and accumulation operations include exponent addition, alignment, mantissa multiplication, channel weight accumulation (CWA) and feature weight accumulation (FWA). CWA and FWA are the processes of accumulating the multiplication results of different channels and different IA bits, respectively. Since CWA must be later than exponent addition and alignment, and FWA is irrelevant to the alignment result, the FWA-first stream is used in the RIMA of the embodiment of the present invention to parallelly calculate exponent addition and alignment as well as mantissa multiplication and FWA. The computational efficiency is significantly better than the computational method of the prior art CWA-first Serial-EM.

[0083] The second embodiment of the present invention discloses a multi-chip module based on a core particle, such as Figure 6 As shown. The multi-chip module is based on the macro implementation of the first embodiment of the present invention. The multi-chip module has a weight stable in-memory computing mode (Weight Stable, WS) and an output stable (Output Stable, OS) in-memory computing mode;

[0084] In the weight fixed in-memory calculation mode, each HFP-CIM array sets the static random access memory cells in the SRAM row to store weight data (WAR=16:0), and sets the external memory access (External memory access) calculation priority to minimize the weight;

[0085] According to the kernel parameters and channel parameter settings of the external artificial intelligence network, the weight data is loaded into the static random access memory cells of the SRAM row;

[0086] The input activation stream is vertically transmitted to each HFP-CIM macro through the input activation memory; in-memory calculation is performed in each HFP-CIM array, and the in-memory calculation result of the previous HFP-CIM array is used as the input of the next HFP-CIM array, and the in-memory calculation result of the last HFP-CIM array is transmitted to the RBL driver as an output signal, so that the partial sum calculated by each HFP-CIM array will be horizontally transmitted and accumulated between chips.

[0087] In the output fixed in-memory computing mode, each HFP-CIM array sets the static random access memory unit in the SRAM row to store weight data or part of the data and / or output activation data, which is determined according to the different feature sizes in different layers and tensor shapes of the external artificial intelligence network. Set the external memory access computing priority to minimize the output activation data; set multiple levels of HFP-CIM array groups, each HFP-CIM array group performs in-memory computing in parallel;

[0088] The weight data is transmitted horizontally in each HFP-CIM array, and the input activation stream is transmitted vertically to each HFP-CIM array; in-memory calculation is performed in each horizontal HFP-CIM array group, and the in-memory calculation result of the last HFP-CIM array group is transmitted to the RBL driver as an output signal. This allows the partial sum of each HFP-CIM macro to be accumulated and stored locally, while the OA is transmitted horizontally between chips.

[0089] In the embodiment of the present invention, since multiple HFP-CIM macros are set in the multi-chip module, only the boundary HFP-CIM macros need to perform external data interaction and consume I / O bandwidth, and the output activation flow transmission between the remaining chips can be realized by partial and output activation receivers and partial and output activation transmitters, which can widely expand the computing resources of the multi-chip module. Compared with the existing weight fixed memory computing solution, the off-chip storage access of the 3×3 multi-chip module in the embodiment of the present invention can be reduced by 88.9% and 45.4% respectively (for ResNet-50 (ImageNet) and ViT-B (ImageNet)).

[0090] The present invention has positive effects in at least the following aspects:

[0091] 1. Compared with the existing floating-point in-memory computing solutions, the present invention supports the efficient HFP format (HFP4 / 6 / 8 / 16) in in-memory computing for the first time, and significantly improves the data throughput by calculating the exponent and mantissa in parallel.

[0092] 2. Due to the reconfigurable weight activation ratio and core-based in-memory computing multi-chip module with scalable heterogeneous data flow, the present invention supports WS and OS in CIM to minimize the off-chip memory access of the multi-chip module (for example, for ResNet-50, the off-chip memory access of a 3×3 multi-chip module is reduced by 9 times).

[0093] 3. Even excluding the power and latency of off-chip memory access, the present invention achieves the highest energy efficiency and throughput density among existing floating-point in-memory computing schemes with comparable precision.

[0094] 4. Compared with the existing digital accelerator supporting partial HFP format, the present invention achieves 3.4 times improvement in energy efficiency and 1.32 times increase in throughput density.

[0095] Those skilled in the art will appreciate that the modules in the devices in the embodiments of the present invention may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments of the present invention may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including corresponding claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including corresponding claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0096] Through the description of the above implementation methods, technical personnel in the relevant field can understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0097] In addition, each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. In particular, for embodiments such as devices and equipment, since they are basically similar to method embodiments, the relevant parts can refer to the partial description of the method embodiments. The embodiments of the devices and equipment described above are merely schematic, wherein the modules, units, etc. described as separate components may or may not be physically separated, that is, they may be located in one place, or they may be distributed in multiple places, such as nodes of a system network. Specifically, some or all of the modules and units may be selected according to actual needs to achieve the purpose of the above-mentioned embodiment scheme. Those skilled in the art can understand and implement it without paying creative labor.

[0098] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0099] In addition, the terms "first", "second", etc. used in the embodiments of the present invention are only used for descriptive purposes and should not be understood as indicating or implying relative importance, or implicitly indicating the number of technical features indicated in the present embodiment. Therefore, the features defined by the terms "first", "second", etc. in the embodiments of the present invention can explicitly or implicitly indicate that the embodiment includes at least one of the features. In the description of the present invention, the word "multiple" means at least two or two or more, such as two, three, four, etc., unless otherwise clearly and specifically defined in the embodiments.

[0100] In the embodiments of the present invention, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element. In addition, components, features, and elements with the same name in different embodiments of the present invention may have the same meaning or different meanings, and their specific meanings need to be determined by their explanation in the specific embodiment or further combined with the context in the specific embodiment.

[0101] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and cannot be construed as limitations of the present invention, and those of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention. Those skilled in the art will readily come to think of other embodiments of the present invention after considering the specification and practicing the present invention. This application is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art that are not disclosed in the present invention. The specification and embodiments are intended to be exemplary only, and the true scope and spirit of the present invention are indicated by the following claims.

Claims

1. A mixed precision floating point digital storage macro, characterized in that: It includes an HFP-CIM array, an input activation memory, a maximum index finder, an addition tree, an FP combiner and a C2C connector; the HFP-CIM array is respectively connected to the input activation memory, the maximum index finder and the addition tree; the FP combiner is respectively connected to the maximum index finder and the addition tree; the C2C connector is connected to the input activation memory, and connects the HFP-CIM array and the FP combiner through a multiplexer.

2. A mixed precision floating point digital storage macro according to claim 1, characterized in that: The C2C connector includes an input activation receiver, an input activation transmitter, a partial and output activation receiver, a partial and output activation transmitter, a weight receiver and a weight transmitter; wherein the input activation receiver and the input activation transmitter are connected to the input activation memory; the weight receiver is connected to the HFP-CIM array via a multiplexer; The weight transmitter is connected to the HFP-CIM array via a multiplexer; the partial and output activation transmitter is connected to the partial and output activation receiver, the HFP-CIM array and the FP combiner via a multiplexer; The input activation receiver is used to write the external input activation stream into the input activation memory; The weight receiver is used to write the external weight stream into the HFP-CIM array; The partial and output activation receiver is used to transmit an external output activation stream to the partial and output activation transmitter; The partial and output activation transmitter is used to output the output activation stream provided by the HFP-CIM array, the FP combiner and the partial and output activation receiver; The input activation transmitter is used to output the input activation stream provided by the input activation memory; The weight transmitter is used to output the weight stream provided by the HFP-CIM array.

3. A mixed precision floating point digital storage macro according to claim 2, characterized in that: The input activation memory includes a plurality of static random access memories for storing input activation streams; the static random access memories transmit the input activation streams to the HFP-CIM array via a multiplexer.

4. A mixed precision floating point digital storage and calculation macro according to claim 2, characterized in that: The HFP-CIM array includes multiple levels, each level includes multiple SRAM rows, RWL decoders, WWL decoders, WBL drivers, RBL drivers, reconfigurable in-memory accumulators, and mantissa alignment modules; In each level, the RWL decoder, the WWL decoder, the WBL driver and the RBL driver are each connected to each SRAM row; the RWL decoder is also connected to the input activation memory; the RBL driver is also connected to the reconfigurable in-memory accumulator, the partial and output activation transmitter and the weight transmitter through a multiplexer; the reconfigurable in-memory accumulator is connected to the maximum exponent finder and the mantissa alignment module; the mantissa alignment module is connected to the maximum exponent finder and the addition tree; The RWL decoder is used to receive the input activation stream of the input activation memory and transmit the input activation stream to each SRAM row for storage; The WBL driver is used to selectively write the bit lines in the SRAM row; The WWL decoder is used to select a word line in the SRAM row for selective writing; The SRAM row is used to store weight data or partial sum data or output activation data according to the writing results of the WBL driver and the WWL decoder and perform in-memory calculation operations to obtain output signals to be output to the RBL driver; The RBL driver is used to receive the output signal of the SRAM row, and transmit the output signal to the reconfigurable in-memory accumulator through the multiplexer; the RBL driver is also used to generate an output data stream, and transmit the output data stream to the partial and output activation transmitter through the multiplexer; the RBL driver is also used to transmit the weight data stream to the weight transmitter through the multiplexer; The reconfigurable in-memory accumulator is used to receive the input activation stream transmitted by the input activation memory and the output signal of the HFP-CIM array, perform parallel exponential addition operation and input activation bit-level accumulation operation, obtain an exponential addition signal and an accumulation signal, and transmit the exponential addition signal to the maximum exponent finder, and transmit the accumulation signal to the mantissa alignment module; The mantissa alignment module is used to perform a mantissa alignment operation on the accumulated signal output by the reconfigurable in-memory accumulator, and obtain an aligned mantissa signal which is transmitted to an addition tree for a channel-level accumulation operation.

5. A mixed precision floating point digital storage and calculation macro according to claim 4, characterized in that: The RWL decoder includes a read enable configuration or an input activation configuration; each of the SRAM rows includes a plurality of static random access memory cells, each of which is divided into an exponent sector and a mantissa sector; the static random access memory cells of the exponent sector perform exponential addition operations, and the static random access memory cells of the mantissa sector perform mantissa multiplication operations.

6. A mixed precision floating point digital storage and calculation macro according to claim 5, characterized in that: The read word lines of the static random access memory cells in the SRAM row are connected via switches for exponent-mantissa ratio; the SRAM rows are connected via switches for floating point precision; In the HFP-CIM array, the ratio of the SRAM row addition sector and the accumulation sector static random access memory unit can be changed by switching control of the exponent-mantissa ratio to adjust the exponent-mantissa ratio of the HFP-CIM array; the SRAM row can be arbitrarily reorganized by switching the floating-point precision to reconstruct the granularity of the HFP-CIM array, thereby supporting arbitrary floating-point precision.

7. A mixed precision floating point digital storage and calculation macro according to claim 4, characterized in that: The reconfigurable in-memory accumulator includes a plurality of full adders and a group of ping-pong SRAMs; the full adders are arranged in a row and connected to the ping-pong SRAMs via a multiplexer; The full adder and ping-pong SRAM are divided into an addition sector and an accumulation sector according to the setting of the HFP-CIM array; the full adders of the addition sector and the accumulation sector, and the ping-pong SRAMs of the addition sector and the accumulation sector are connected through a switch for the exponent-mantissa ratio; the full adder of the addition sector performs an exponential addition operation, obtains an exponential addition signal, and outputs it to the maximum exponent finder to search for the maximum exponent; the ping-pong SRAM of the accumulation sector cyclically performs read and write operations and cooperates with the full adder of the accumulation sector to perform an accumulation operation of the mantissa at the input activation bit level, and obtains an accumulation signal, which is output to the mantissa module.

8. A mixed precision floating point digital storage and calculation macro according to claim 7, characterized in that: The full adder of the addition sector receives the output signal from the RBL driver transmission SRAM row and the exponential signal of the input data stream, performs exponential addition operation, and outputs the exponential addition signal to the exponential sector of the ping-pong SRAM; The full adder of the accumulation sector receives the output signal from the RBL driver transmission SRAM row and the mantissa accumulation partial sum signal stored in one row of the accumulation sector ping-pong SRAM in the previous cycle, performs mantissa accumulation operation, and outputs the accumulated partial sum to another row of the ping-pong SRAM.

9. A multi-chip module based on a chiplet, characterized in that: The multi-chip module has a macro as claimed in any one of claims 4 to 8, and the multi-chip module has a weight fixed in-memory calculation mode and an output fixed in-memory calculation mode; Among them, the weight fixed in-memory calculation mode specifically includes: Setting the static random access memory cells in the SRAM row to store weight data, and setting the external memory access calculation priority that minimizes the weight; According to the kernel parameter and channel parameter settings, the weight data is loaded into the static random access memory cells of the SRAM row; The input activation stream is vertically transmitted to each HFP-CIM macro through the input activation memory; in-memory calculation is performed in each HFP-CIM array, the in-memory calculation result of the previous HFP-CIM array is used as the input of the next HFP-CIM array, and the in-memory calculation result of the last HFP-CIM array is transmitted to the RBL driver as an output signal; Output fixed memory calculation modes include: Setting the static random access memory cells in the SRAM row to store weight data or partial data and / or output activation data, and setting an external memory access calculation priority that minimizes output activation data; Setting multiple horizontal HFP-CIM array groups, each HFP-CIM array group performs in-memory calculations in parallel; The weight data is horizontally transmitted in each HFP-CIM macro, and the input activation stream is vertically transmitted to each HFP-CIM macro; in-memory calculation is performed in each horizontal HFP-CIM array group, and the in-memory calculation result of the last HFP-CIM array group is transmitted to the RBL driver as an output signal.

10. The multi-chip module based on core particles according to claim 9, characterized in that: The external memory access calculation priority for minimizing weights and / or the external memory access calculation priority for minimizing output activation data are determined based on the external artificial intelligence model structure of the external input data stream source.

Images