Method for realizing LLM dynamic and static KV Cache pruning based on UniCAIM architecture

Abstract

The invention discloses a method for realizing LLM dynamic and static KV Cache pruning based on a UniCAIM architecture, and belongs to the technical field of novel storage and calculation. The invention provides a unified in-memory calculation and in-memory search (UniCAIM) architecture, a UniCAIM unit in the UniCAIM architecture is designed based on FeFET, during each step of decoding, the most relevant token is dynamically selected based on a CAM mode to participate in accurate attention calculation, the calculation overhead is reduced, and when the generation length exceeds the reserved KV Cache capacity, the accumulated attention score is calculated based on the CIM mode to statically reject the token, so that the calculation efficiency is improved. And through dynamic and static KV Cache pruning, the storage and calculation overhead is reduced at the same time. According to the method, the area-energy-delay product is reduced at the circuit level, meanwhile, the precision equivalent to dense attention calculation is kept at the application level, and the method has huge potential in long context LLM reasoning.

Description

Technical Field

[0001] The present invention relates to the field of novel storage and computing technologies, and in particular to a method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture. Background Art

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) based on the Transformer architecture have made breakthrough progress in the field of Natural Language Processing (NLP) and are widely used in tasks such as dialogue systems, content generation, intelligent search, and document summarization. Typical LLMs, such as GPT and LLaMA, rely on the self-attention mechanism to globally model all tokens in the input sequence to achieve contextual understanding and semantic reasoning. When the sequence length is n, the computational complexity of the self-attention mechanism is O(n 2 To reduce computational overhead, researchers have proposed the Key-Value Cache (KV cache) technology. This technology saves the key and value of historical tokens to avoid repeatedly calculating the attention weights of historical contexts in each generation round, thereby accelerating the speed of multiple rounds of generation. However, this technology incurs greater storage overhead. In particular, as model depth increases and sequence length expands, the capacity requirements of the KV Cache increase linearly. The calculation of self-attention has also become a key bottleneck limiting the deployment and response speed of LLM. In addition, KV Cache accesses incur frequent data transmission overhead and require a large number of matrix-vector multiplication operations to calculate attention scores, resulting in higher latency and power consumption, which is particularly prominent on edge devices.

[0003] To address these issues, researchers have proposed different solutions from both algorithmic and hardware perspectives. Algorithmically, researchers have proposed KV Cache pruning strategies, including static and dynamic pruning. Static pruning permanently removes unimportant tokens from the KV Cache by pre-defining important locations or based on accumulated attention scores. Dynamic pruning selects tokens with the highest attention scores based on the current query during autoregressive generation. However, static pruning lacks query-specific adaptability and is prone to loss of contextual information, resulting in reduced accuracy. While dynamic pruning offers greater flexibility, it requires more complex computations, such as attention score sorting and selection, posing greater challenges to hardware implementation. On the hardware side, researchers have used a compute-in-memory (CIM) architecture to accelerate the matrix-vector multiplication operations in attention calculations, reducing data movement overhead and improving computational efficiency. They have also introduced KV Cache pruning strategies, but these only support fixed static or dynamic pruning, resulting in reduced accuracy and limited computational efficiency. Therefore, collaborative design of software and hardware is required, which can combine KV Cache dynamic and static pruning technology and in-memory computing architecture to significantly reduce storage requirements and improve inference energy efficiency while maintaining model accuracy, providing a feasible solution for edge-side long text LLM inference. Summary of the Invention

[0004] To address the above problems in the existing technologies, the present invention proposes a method for implementing dynamic and static key-value cache (KV Cache) pruning of large language models (LLMs) based on a unified in-memory computation and in-memory search (UniCAIM) architecture. The method can achieve efficient dynamic KV cache pruning with O(1) time complexity, as well as flexible static KV cache pruning based on accumulated similarity scores. A hardware implementation based on ferroelectric field-effect transistors (FeFETs) is designed, which reduces the area-energy-delay product at the circuit level while maintaining the same accuracy as dense attention calculation at the application level. It has great potential in long-context LLM reasoning.

[0005] The technical solutions of the present invention are as follows:

[0006] A method for implementing LLM dynamic and static KV Cache pruning based on UniCAIM architecture includes the following steps:

[0007] 1) The UniCAIM array uses a NOR type array based on the UniCAIM unit. The UniCAIM unit is composed of two storage cells for storing signed keys and implementing local signed multiplication calculations of keys and queries. Each storage cell is composed of a field effect transistor MOSFET and a FeFET. The source of the MOSFET is connected to the gate of the FeFET. The two storage cells are connected in parallel, that is, the gates of the two MOSFETs are connected to form WL, the drains of the two FeFETs are connected to form SL, and the drains of the two MOSFETs are BL and BL respectively. b , the sources of the two FeFETs are grounded;

[0008] 2) Each row of the UniCAIM array shares a dynamic pruning circuit and a static pruning circuit. The dynamic pruning circuit includes a precharge transistor connected to the SL and a buffer. The precharge transistor and the buffer are connected in series. The output of the buffer is connected to the gate of a FeFET programmed to an appropriate threshold voltage. Its on-current is I0. The FeFETs in each row are connected in parallel to obtain an accumulated current I1, which is added to the reference current I Ref1 Connected to the two inputs of the comparator, the output of the comparator is the control signal Ctrl1, and the outputs of all buffers are connected to the local register; the static pruning circuit includes a switch S1 connected to SL, an accumulation capacitor, a discharge transistor and an adjustable inverter, the switch S1 and the adjustable inverter are connected in series, the accumulation capacitor and the discharge transistor are connected in parallel and connected between S1 and the adjustable inverter, the adjustable inverter is composed of an n-type FeFET and a p-type FeFET according to the structure of an inverter, the output of the adjustable inverter is connected to the gate of a FeFET programmed to an appropriate threshold voltage, and its on-current is I0, and each row of FeFETs is connected in parallel to obtain an accumulated current I2, which is compared with the reference current I Ref2 Connected to the two inputs of the comparator, the output of the comparator is the control signal Ctrl2, and the outputs of all adjustable inverters are connected to the local register;

[0009] 3) In the dynamic pruning stage, the UniCAIM architecture works in the content addressable memory CAM mode. The content addressable memory first charges all SLs to VDD and floats them through the precharge transistors, and then biases the BL to the corresponding voltage according to the input query. By setting I Ref1 (k+1)I0, when k SLs with higher similarity maintain a high level, the accumulated current I1 is less than I Ref1, causing Ctrl1 to flip, stopping the discharge of SL, selecting the k keys with the highest similarity to the query, and storing the dynamically selected k keys in the local register for subsequent accurate attention calculation. After dynamic KV Cache pruning, the UniCAIM architecture works in the charge domain CIM mode, closing the switch S1, and the charge on the SL is shared with the accumulation capacitor. Then, the switch S1 is opened, and the charge on the accumulation capacitor is discharged through the discharge transistor, adjusting the flip voltage V of the adjustable inverter. S , the voltage on the accumulation capacitor corresponding to the key with the smallest accumulated attention score will be discharged to V first S , so that the output voltage of its corresponding adjustable inverter is flipped to a high voltage, turning on the FeFET connected to it, making I2 greater than I set to I0 Ref2 , causing Ctrl2 to flip, turning off the discharge transistor and storing the address of the key with the lowest accumulated similarity in the local register;

[0010] 4) All SLs in the UniCAIM array are connected to an N-to-k multiplexer and k analog-to-digital converters (ADCs) through switch S2. After completing dynamic and static KV Cache pruning, the UniCAIM architecture operates in current domain CIM mode, closing switch S2 to apply a fixed voltage to the SLs where the k dynamically selected keys are located. The precise attention scores are quantified by the ADC to complete the precise attention calculation. When the number of generated keys exceeds the storage capacity, the key with the lowest accumulated similarity is statically eliminated.

[0011] Furthermore, the threshold voltages of the FeFETs of the two UniCAIM units are programmed to complementary states, i.e., high threshold voltage and low threshold voltage, indicating that the storage key is "-1" or "+1". b Applying complementary high and low read voltages indicates that the query is "-1" or "+1", and the high and low currents of SL indicate that the calculation results are "-1" and "+1".

[0012] Furthermore, both FeFETs of the UniCAIM unit are programmed to a medium threshold voltage state, indicating that the storage key is “0”. For different queries, the SL current is medium, indicating that the calculation result is “0”.

[0013] Furthermore, the ferroelectric material of the FeFET needs to adopt various HfO2-doped multi-domain ferroelectric materials such as HfO2 doped with Zr (HZO) and HfO2 doped with Al (HfAlO), and the device gate stack can be based on various structures such as MFMIS, MFIS, and MFS.

[0014] The technical effects of the present invention are as follows:

[0015] 1. The present invention proposes a method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture. At each decoding step, the most relevant token is dynamically selected to participate in the precise attention calculation to reduce the computational overhead. When the generated length exceeds the reserved KV Cache capacity, the token is statically eliminated based on the accumulated attention score to improve memory utilization. Through dynamic and static KV Cache pruning, storage and computational overhead are reduced at the same time.

[0016] 2. The present invention proposes a method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture. By utilizing the CAM mode of UniCAIM, attention scores can be approximately evaluated and compared. Dynamic KV Cache pruning can be achieved in O(1) time without precise calculation, thereby reducing computing power consumption. Furthermore, the charge domain CIM is used to implement the evaluation of accumulated attention scores in the same operation cycle for static KV Cache pruning, thereby improving memory utilization. Finally, the current domain CIM is used to implement accurate attention score calculation to ensure calculation accuracy.

[0017] 3. This paper proposes a method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture, which reduces the area-energy-delay product at the circuit level while maintaining the same accuracy as dense attention calculation at the application level, and has great potential in long-context LLM reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a schematic diagram of the UniCAIM architecture proposed in the present invention;

[0019] Figure 2 This is a schematic diagram of the circuit implemented based on FeFET hardware based on the UniCAIM architecture proposed in the present invention. DETAILED DESCRIPTION

[0020] The present invention will be further clearly and completely described below through specific embodiments in conjunction with the accompanying drawings.

[0021] The UniCAIM architecture proposed in this invention includes UniCAIM array and dynamic KV Cache pruning, static KV Cache pruning and attention calculation. Figure 1As shown, the UniCAIM array is used to store key vectors, including important tokens generated and retained in the pre-filling phase, as well as newly generated tokens for the autoregressive decoding phase. In each decoding step, dynamic KV Cache pruning is performed by evaluating the approximate attention score, and k tokens with the highest attention scores are selected for subsequent precise attention calculations. In addition, static KV Cache pruning is performed by evaluating the accumulated attention score. When the number of tokens exceeds the storage capacity of the UniCAIM array, the token with the lowest accumulated attention score is selected and permanently eliminated. Through dynamic and static KVCache pruning, UniCAIM can maintain a fixed array size, achieve higher storage utilization and reduce the overhead of precise attention calculations.

[0022] Figure 2 This is a circuit diagram of the FeFET hardware implementation of the UniCAIM architecture proposed in this invention, including a FeFET-based UniCAIM array and peripheral circuits for dynamic and static pruning. The UniCAIM array is a NOR-type array based on UniCAIM cells. The UniCAIM cell consists of two MOSFETs and two FeFETs and can store signed keys and perform local signed multiplication of keys and queries. The specific operation steps include:

[0023] 1) The UniCAIM array (N rows) is used to store key vectors, which consists of two parts: one is the important word units (tokens) generated and retained in the pre-filling stage, and the other is the newly generated tokens used in the autoregressive decoding stage. In each decoding step, dynamic KV Cache pruning is performed by evaluating the approximate attention score, and k tokens with the highest attention scores are selected for subsequent precise attention calculations. In addition, static KV Cache pruning is performed by evaluating the accumulated attention score. When the number of tokens exceeds the storage capacity of the UniCAIM array, the token with the lowest accumulated attention score is selected and permanently eliminated. Through dynamic and static KV Cache pruning, UniCAIM can maintain a fixed array size, achieve higher storage utilization and reduce the overhead of precise attention calculations.

[0024] The UniCAIM array is a NOR type array based on the UniCAIM unit. The UniCAIM unit is composed of two FeFET memory cells. Each FeFET memory cell is composed of a field effect transistor (MOSFET) and a FeFET. The source of the MOSFET is connected to the gate of the FeFET. The two memory cells are connected in parallel, that is, the gates of the two MOSFETs are connected to form WL, the drains of the two FeFETs are connected to form SL, and the drains of the two MOSFETs are BL and BL respectively. b , the sources of the two FeFETs are grounded. The UniCAIM unit can store signed keys and implement local signed multiplication calculations of key and query, i.e. local attention score calculations. By programming the threshold voltages of the two FeFETs to complementary states, i.e. high threshold voltage and low threshold voltage, the stored key can be represented as "-1" or "+1". By b Applying complementary high and low read voltages indicates that the query is "-1" or "+1." The result of the signed multiplication can be obtained from the current of the SL, with high and low currents indicating the calculation results of "-1" and "+1." In addition, by programming both FeFETs to a medium threshold voltage state, the storage key can be represented as "0." For different queries, the SL current is medium, indicating the calculation result is "0." Further leveraging the multi-value storage capability of the FeFET, programming the threshold voltages of the two FeFETs to multiple different complementary states can achieve in-situ multi-value expansion of the signed key. The multi-value expansion of the signed query is achieved through bitwise expansion.

[0025] 2) The UniCAIM architecture can work in content addressable memory (CAM) mode, that is, in-memory search mode, to complete dynamic KV Cache pruning. Each row of the UniCAIM array shares a dynamic pruning circuit, including a precharge transistor connected to the SL and a buffer. The precharge transistor and the buffer are connected in series. The output of the buffer is connected to the gate of a FeFET programmed to an appropriate threshold voltage (whose conduction current is I0). The FeFETs in each row are connected in parallel to obtain an accumulated current I1, which is compared with the reference current I Ref1Connected to the two inputs of the comparator, the output of the comparator is the control signal Ctrl1. In the dynamic pruning stage, all SLs are first charged to a high voltage (VDD) through the pre-charge transistor and left floating. Then, the BL is biased to the corresponding voltage according to the input query. Based on the working principle of the UniCAIM unit, the key with a higher similarity to the query will have a smaller SL current, so that the SL has a slower discharge speed. Conversely, the key with a lower similarity to the query will have a faster discharge speed, causing the SL to discharge faster to below VDD / 2, causing the output of the buffer to become a low voltage and the FeFET connected to it to be turned off. By setting I Ref1 =(k+1)I0, we can select the k keys with the highest similarity to the query. When the k SLs with higher similarity maintain a high level, the accumulated current I1 will be less than I Ref1 , causing Ctrl1 to flip, stopping the SL discharge process, and storing the dynamically selected k keys in a local register for subsequent accurate attention calculation. The UniCAIM-based CAM mode can complete dynamic KV Cache pruning with O(1) time complexity and does not require actual similarity calculation. In addition, by programming the threshold of the FeFET in the dynamic pruning circuit, the number of dynamically selected tokens, that is, the size of k, can be flexibly configured.

[0026] 3) The UniCAIM architecture can work in charge domain in-memory computing (CIM) mode to complete static KV Cache pruning. Each row of the UniCAIM array shares a static pruning circuit. The static pruning circuit includes a switch S1 connected to SL, and the switch S1 is connected to the accumulation capacitor, the discharge transistor and an adjustable inverter. The switch S1 and the adjustable inverter are connected in series with each other. The accumulation capacitor and the discharge transistor are connected in parallel and connected between S1 and the adjustable inverter. The adjustable inverter is composed of n-type FeFET and p-type FeFET according to the inverter structure. The output of the adjustable inverter is also connected to the gate of a FeFET programmed to an appropriate threshold voltage (whose on-current is I0). The FeFETs in each row are connected in parallel to obtain an accumulated current I2, which is compared with the reference current I Ref2 Connected to the two inputs of the comparator, the output of the comparator is the control signal Ctrl2. After the dynamic KV Cache pruning, the switch S1 is closed, and the charge on the SL is shared with the accumulation capacitor. Then the switch S1 is opened, and the charge on the accumulation capacitor is discharged through the discharge transistor, adjusting the flip voltage of the adjustable inverter (V S ), the voltage on the accumulation capacitor corresponding to the key with the smallest accumulated attention score will be discharged to V first S, so that the output voltage of its corresponding adjustable inverter is flipped to a high voltage, turning on the FeFET connected to it, making I2 greater than I set to I0 Ref2 , causing Ctrl2 to flip, turning off the discharge transistor. The address of the key with the lowest accumulated similarity is stored in a local register. After the precise attention calculation is completed, the key of the newly generated token is directly written to this address, completing the static KV Cache pruning and maintaining a fixed capacity of the UniCAIM array.

[0027] 4) The UniCAIM architecture can operate in current-domain CIM mode. All SLs in the UniCAIM array are connected to an N-to-k multiplexer (MUX) and then to k analog-to-digital converters (ADCs) via switch S2. After completing dynamic and static KV Cache pruning, switch S2 is closed, applying a fixed voltage to the SLs containing the dynamically selected k keys. The ADC quantizes the precise attention scores. Due to dynamic KV Cache pruning, only a portion of the attention score needs to be quantized by the ADC, reducing computational power consumption. Furthermore, since the carefully designed FeFET-based UniCAIM unit operates on the principle that the greater the attention score, the smaller the SL current. Therefore, dynamically selected tokens that require precise calculation correspond to smaller SL currents, further reducing the power consumption of attention calculations.

[0028] This embodiment fully and in detail describes the method for implementing LLM dynamic and static KV Cache pruning in the UniCAIM architecture. The UniCAIM architecture of the present invention can work in CAM mode. In the dynamic pruning stage, all SLs are first charged to VDD and floated by pre-charge transistors. Then, according to the input query, the BL is biased to the corresponding voltage. Keys with higher similarity to the query will have smaller SL currents, so that the SL has a slower discharge speed. By setting I Ref1 =(k+1)I0, we can select the k keys with the highest similarity to the query and store them in the local register for subsequent accurate attention calculation. The UniCAIM architecture can work in charge domain CIM mode. After dynamic KV Cache pruning, the switch S1 is closed, and the charge on SL is shared with the accumulation capacitor. Then the switch S1 is opened, and the charge on the accumulation capacitor is discharged through the discharge transistor, adjusting the V of the adjustable inverter. S , the voltage on the accumulation capacitor corresponding to the key with the smallest accumulated attention score will be discharged to V first S , so that I2 is greater than I set to I0 Ref2The key with the lowest accumulated similarity will be statically eliminated after the precise attention calculation is completed. The UniCAIM architecture can operate in current-domain CIM mode. After completing dynamic and static KV Cache pruning, switch S2 is closed to apply a fixed voltage to the SL where the dynamically selected k keys are located, and the precise attention score is quantified by the ADC.

[0029] The present invention implements efficient dynamic KV cache pruning with O(1) time complexity based on the CAM mode, implements flexible static KV cache pruning by accumulating similarity scores based on the charge domain CIM, while reducing storage and computational overheads, and finally uses the current domain CIM to achieve accurate attention score calculation to ensure calculation accuracy.

[0030] In addition, the UniCAIM unit proposed in the present invention can also be designed based on other memory cells. It only needs to replace the storage unit from FeFET with other storage elements such as floating gate transistor flash memory (Flash), resistive random access memory (RRAM), phase change memory (PCM), etc.

[0031] Finally, it should be noted that the purpose of disclosing the embodiments is to facilitate a further understanding of the present invention. However, those skilled in the art will appreciate that various substitutions and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the contents disclosed in the embodiments, and the scope of protection claimed by the present invention shall be determined by the scope defined in the claims.

Claims

1. A method for implementing LLM dynamic and static KV Cache pruning based on UniCAIM architecture, comprising the following steps: 1) The UniCAIM array uses a NOR type array based on the UniCAIM unit. The UniCAIM unit is composed of two storage cells for storing signed keys and implementing local signed multiplication calculations of keys and queries. Each storage cell is composed of a field effect transistor MOSFET and a FeFET. The source of the MOSFET is connected to the gate of the FeFET. The two storage cells are connected in parallel, that is, the gates of the two MOSFETs are connected to form WL, the drains of the two FeFETs are connected to form SL, and the drains of the two MOSFETs are BL and BL respectively. b , the sources of the two FeFETs are grounded; 2) Each row of the UniCAIM array shares a dynamic pruning circuit and a static pruning circuit. The dynamic pruning circuit includes a precharge transistor connected to the SL and a buffer. The precharge transistor and the buffer are connected in series. The output of the buffer is connected to the gate of a FeFET programmed to an appropriate threshold voltage. Its on-current is I0. The FeFETs in each row are connected in parallel to obtain an accumulated current I1, which is added to the reference current I Ref1 Connected to the two inputs of the comparator, the output of the comparator is the control signal Ctrl1, and the outputs of all buffers are connected to the local register; the static pruning circuit includes a switch S1 connected to SL, an accumulation capacitor, a discharge transistor and an adjustable inverter, the switch S1 and the adjustable inverter are connected in series, the accumulation capacitor and the discharge transistor are connected in parallel and connected between S1 and the adjustable inverter, the adjustable inverter is composed of an n-type FeFET and a p-type FeFET according to the structure of an inverter, the output of the adjustable inverter is connected to the gate of a FeFET programmed to an appropriate threshold voltage, and its on-current is I0, and each row of FeFETs is connected in parallel to obtain an accumulated current I2, which is compared with the reference current I Ref2 Connected to the two inputs of the comparator, the output of the comparator is the control signal Ctrl2, and the outputs of all adjustable inverters are connected to the local register; 3) In the dynamic pruning stage, the UniCAIM architecture works in the content addressable memory CAM mode. The content addressable memory first charges all SLs to VDD and floats them through the precharge transistors, and then biases the BL to the corresponding voltage according to the input query. By setting I Ref1 (k+1)I0, when k SLs with higher similarity maintain a high level, the accumulated current I1 is less than I Ref1 , causing Ctrl1 to flip, stopping the discharge of SL, selecting the k keys with the highest similarity to the query, and storing the dynamically selected k keys in the local register for subsequent accurate attention calculation. After dynamic KV Cache pruning, the UniCAIM architecture works in the charge domain CIM mode, closing the switch S1, and the charge on the SL is shared with the accumulation capacitor. Then, the switch S1 is opened, and the charge on the accumulation capacitor is discharged through the discharge transistor, adjusting the flip voltage V of the adjustable inverter. S , the voltage on the accumulation capacitor corresponding to the key with the smallest accumulated attention score will be discharged to V first S , so that the output voltage of its corresponding adjustable inverter is flipped to a high voltage, turning on the FeFET connected to it, making I2 greater than I set to I0 Ref2 , causing Ctrl2 to flip, turning off the discharge transistor and storing the address of the key with the lowest accumulated similarity in the local register; 4) All SLs in the UniCAIM array are connected to an N-to-k multiplexer and k analog-to-digital converters (ADCs) through switch S2. After completing dynamic and static KV Cache pruning, the UniCAIM architecture operates in current domain CIM mode, closing switch S2 to apply a fixed voltage to the SLs where the k dynamically selected keys are located. The precise attention scores are quantified by the ADC to complete the precise attention calculation. When the number of generated keys exceeds the storage capacity, the key with the lowest accumulated similarity is statically eliminated.

2. The method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture according to claim 1, characterized in that: The threshold voltages of the FeFETs of the two UniCAIM units are programmed to complementary states, i.e., high threshold voltage and low threshold voltage, indicating that the storage key is "-1" or "+1". b Applying complementary high and low read voltages indicates a query of "-1" or "+1", and the high and low currents of the SL indicate calculated results of "-1" and "+1".

3. The method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture as claimed in claim 1, characterized in that: Both FeFETs in the UniCAIM cell are programmed to a medium threshold voltage state, indicating that the storage key is "0". For different queries, the SL current is medium, indicating that the calculation result is "0".

4. The method for implementing LLM dynamic and static KV Cache pruning based on the UniCAIM architecture as claimed in claim 1, characterized in that: The ferroelectric material of the FeFET is HfO2 doped with Zr or HfO2 doped with Al.

5. The method for implementing LLM dynamic and static KV Cache pruning based on UniCAIM architecture according to claim 1, characterized in that: The gate stack of the FeFET is based on MFMIS, MFIS or MFS structure.

Images