Artificial intelligence computing system and data processing method thereof, in-memory computing device

Abstract

The application relates to a data processing method and an in-memory computing device of an artificial intelligence computing system, comprising the following steps: a data format conversion step: acquiring original data in a first data format, and converting the original data into a second data format of in-memory computing CIM array hardware characteristics by using a format conversion unit; a data synchronous transmission step: shunting data in the second data format into a first path and a second path for transmission; transmitting the data in the second data format to the in-memory computing CIM array through the first path; and storing the data in the second data format as a formatted data copy through the second path, so that the original data can be directly called without format conversion when the artificial intelligence computing system needs the original data again, and the "one-time data format conversion and multiple data reuse" are realized. Through synchronous data shunting after format conversion, real-time calculation and data backup are completed, and the algorithm and power consumption overhead caused by repeated preprocessing such as quantization and transposition of the same original data in subsequent reasoning is eliminated.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, specifically to an artificial intelligence computing system and its data processing method, and an in-memory computing device. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely applied in fields such as natural language processing, multi-turn dialogue, and content generation. The reasoning process of large language models is typically divided into a prefill stage and a decoding stage. To improve computational efficiency and reduce memory access frequency, existing technologies often employ key-value caching (KV Cache) technology, storing the key-value pairs of previously generated tokens in external memory for reuse when generating subsequent tokens.

[0003] However, in practical applications, due to the physical constraints of the underlying computing hardware (such as in-memory computing arrays, dedicated AI accelerators, etc.), raw format data stored in external memory often cannot directly participate in matrix operations. For example, in-memory computing (CIM) arrays typically require input data to be in a specific fixed-point format (such as INT8 / INT4), and the physical topology often requires the data to meet a specific arrangement (such as column-major or transposed layout).

[0004] Therefore, in traditional large-model inference processes, whenever the in-memory CIM array needs to access historical key-value caches or model weights, it must first retrieve the raw data from external memory. The hardware conversion unit then performs repeated quantization, transpose, or rearrangement operations to convert this data into a format adapted to the underlying hardware characteristics. In scenarios involving high-frequency data reuse, such as long-sequence dialogues, batch inference, or speculative decoding, this "read-every-time, convert-every-time" approach generates significant redundant computational overhead.

[0005] Specifically, repetitive format conversions not only significantly increase the first-word latency (TTFT) and per-token generation latency (TPOT) of inference, but also lead to substantial dynamic power consumption waste, becoming a core bottleneck restricting the deployment of large models on resource-constrained devices or in high-throughput cloud environments. Therefore, optimizing data flow control during inference and reducing redundant preprocessing overhead has become a pressing technical problem in the field of artificial intelligence computing. Summary of the Invention

[0006] To address the aforementioned technical problems, this application provides a data processing method applied to an artificial intelligence computing system, comprising: Data format conversion steps: Obtain the original data in the first data format, and use the format conversion unit to convert it into a second data format that is compatible with the hardware characteristics of the in-memory computing CIM array of the artificial intelligence computing system; Data synchronization transmission steps: Transmit the data in the second data format synchronously along the first path and the second path respectively; Specifically, the second data format data is transmitted to the in-memory computing (CIM) array via the first path to perform computing tasks; the second data format data is stored as a formatted data copy via the second path so that the artificial intelligence computing system can directly call upon the original data when needed again.

[0007] The data processing method provided in this application achieves "one-time data format conversion, multiple data reuses." By synchronously splitting the data after format conversion and simultaneously completing "real-time computation" and "formatted data copy construction," the computational and power consumption overhead caused by repeatedly performing preprocessing such as quantization and transposition on the same original data in subsequent inference is eliminated at the source. Optionally, the data processing method also includes: In the data reuse step, when it is detected that the computing task needs to use the original data again, the formatted data copy corresponding to the original data is read directly.

[0008] This design achieves a highly simplified and optimized computational pipeline. When the original data needs to be reused, the complex preprocessing steps are skipped by directly reading the formatted copy of the original data, significantly reducing processing latency.

[0009] Optionally, the data reuse step further includes: bypassing / disabling the format conversion unit in response to a computing task requiring the use of the original data again.

[0010] This design reduces the power consumption of the AI ​​computing system by bypassing / disabling the format conversion unit. Furthermore, to further reduce the power consumption of the AI ​​computing system, a low-power mode can be entered when the format conversion unit is bypassed / disabled, reducing its operating voltage or cutting off its power supply.

[0011] Optionally, the step of "storing the data of the second data format as a formatted data copy through the second path for direct retrieval by the artificial intelligence computing system when the original data is needed again" includes: selectively storing the data of the second data format in an external memory or the in-memory computing CIM array according to the data size, access frequency and life cycle.

[0012] Specifically, the size of the data, the frequency of access, and the length of the lifespan can be dynamically determined based on hardware conditions or scenario task requirements. The basic selection logic for storing data in external memory or the in-memory computing CIM array is as follows: large-scale data, low-frequency access, or long-lifespan data are preferentially stored in external memory; small-scale data, high-frequency access, or short-lifespan data (burst reuse within a single task) are preferentially stored in the in-memory computing CIM array.

[0013] This design enables AI computing systems to achieve a dynamic balance between "high-capacity storage" and "ultimate reusability performance" based on the scale, frequency of access, and lifecycle of the data.

[0014] Optionally, if the original data is key-value cache data of a large language model, a formatted copy of the converted data is stored in the external memory; the formatted data copy is directly accessed through the external memory in subsequent inference rounds or batch inference tasks to support long-lifecycle data reuse.

[0015] For large language model key-value caches (KV Cache) where the data volume grows linearly with the dialogue length, the large capacity of external memory (such as DRAM) is utilized to solve the physical limitation that on-chip space is insufficient to carry the full amount of historical data in long text (such as 128K context) and multi-turn dialogue scenarios.

[0016] Optionally, if the original data is neural network model weight data or speculative decoding verification feature data, then a formatted copy of the converted data is stored in the in-memory computing CIM array to support zero-latency reading through parallel multiply-accumulate operations in the data reuse step.

[0017] For critical data that is frequently accessed, such as model weights or speculative decoding, the storage and computing integration of the CIM array is utilized. Data copies reside directly within the in-memory computing CIM array where the computation takes place, in a format adapted to the hardware layout (such as column-first arrangement). This eliminates expensive external memory access operations, realizes "data pass-through loading", and further reduces latency.

[0018] Optionally, the raw data includes key-value cache data of a large language model. The data in the first data format is the key-value pair generated and calculated by the large language model for the current token. It is quantized or transposed into formatted key-value data in the second data format through the data format conversion step. The formatted key-value data is transmitted to the in-memory computing CIM array through the first path through the data synchronization transmission step to perform computing tasks. It is stored as a copy of the formatted key-value data through the second path for direct access by the artificial intelligence computing system.

[0019] This design solves the performance bottleneck of long sequence inference. Addressing the significant KV cache conversion pressure in large model inference, it achieves "zero-conversion" fast reading of historical token key-value pairs by storing formatted data copies, greatly improving the speed of long text generation.

[0020] Optionally, the original data includes large language model weight data. During the pre-filling stage of the large language model, the data format conversion step is performed, converting the model weight data from a first data format to a second data format. After the data format conversion is completed, a data synchronization transmission step is performed, transmitting the data in the second data format to the in-memory computing CIM array to participate in the pre-filling calculation, and then transmitting it to an external memory to store it as a formatted data copy. During the decoding stage, a data reuse step is performed, whereby the in-memory computing CIM array directly reads the formatted data copy from the external memory to perform the calculation task.

[0021] This design solves the performance bottleneck of long sequence inference. Addressing the significant KV cache conversion pressure in large model inference, it achieves "zero-conversion" fast reading of historical token key-value pairs by storing formatted copies, greatly improving the speed of long text generation.

[0022] Optionally, the data processing method is applied to cloud-based batch inference scenarios: For each user request in batch inference, its key-value cache data is only converted and a formatted copy is created once during generation; When the key-value cached data is subsequently scheduled or used for inference, data reuse is achieved by reading the corresponding formatted data copy.

[0023] Optionally, the data processing method is applied to multimodal large model inference scenarios: For image feature key value cache data, text feature key value cache data, or intermediate feature data in the diffusion model, data format conversion is performed only once during the first generation, and a formatted data copy is created and written to external storage. In subsequent iterative steps or cross-attention calculation stages, the calculation task is performed by reading the formatted data copy.

[0024] Optionally, the data processing method is applied to speculative decoding acceleration scenarios: Historical key-value cache data undergoes a data format conversion only once during generation; In the verification process of the large model verification draft token, the formatted data copy is read directly to eliminate the repeated format conversion processing of historical key-value cache in the verification process.

[0025] Optionally, the in-memory computing CIM array is an in-memory computing array; the first data format is a floating-point format; the second data format is a fixed-point format data that matches the physical topology of the in-memory computing array, including fixed-point mantissa and sharing index.

[0026] This design balances computational accuracy with storage compression. It employs a shared exponent format adapted to the CIM topology, leveraging the high energy efficiency of fixed-point computing while preserving near-floating-point dynamic range and accuracy through exponent sharing.

[0027] To achieve the above-mentioned objectives, this application provides an in-memory computing device, comprising: In-memory computing (CIM) arrays are used to perform computational tasks required for neural network inference. The format conversion unit is configured to be connected to an external memory and is used to convert the original data from a first data format to a second data format adapted to the hardware characteristics of the in-memory computing CIM array; A data stream splitter controller is connected to the format conversion unit, the external memory, and the in-memory computing CIM array, respectively, and is used to transmit the data in the second data format output by the format conversion unit via a first path and a second path, wherein the data is transmitted to the in-memory computing CIM array via the first path and to the external memory via the second path to be stored as a formatted data copy. The instruction controller is connected to the in-memory computing CIM array, the format conversion unit, and the data stream splitter controller, respectively. It can respond to the computing request of the in-memory computing CIM array, read the formatted data copy, and issue instructions to bypass or disable the format conversion unit.

[0028] The in-memory computing device provided in this application balances computational accuracy and storage compression ratio. It adopts a shared exponent format adapted to CIM topology, which not only utilizes the high energy efficiency of fixed-point computing power, but also retains near-floating-point dynamic range and accuracy through exponent sharing.

[0029] Optionally, the in-memory computing device further includes: An external storage interface is used to connect the external storage and the format conversion unit to enable the reading and writing of the original data and the formatted data copy.

[0030] The external storage interface ensures efficient and compatible data exchange. Managing raw data and formatted copies through a unified interface simplifies the addressing and access logic of external storage.

[0031] Optionally, the instruction controller is configured to: at the initial inference time of the neural network, control the data stream splitter controller to enable the data splitting transmission function; and when the formatted data copy needs to be called later, control the format conversion unit to enter the off or bypass state, and directly call the formatted data copy from the external memory and transmit it to the in-memory computing CIM array.

[0032] This design enables fully automated energy efficiency management. It can automatically switch between "split mode" and "bypass mode" based on inference time, allowing the hardware to achieve optimal efficiency balance without deep software intervention.

[0033] Optionally, the in-memory computing device further includes a format conversion unit bypass switch, which is connected in parallel with the format conversion unit and connected to the instruction controller, and is capable of bypassing the format conversion unit in response to an instruction from the instruction controller.

[0034] The format conversion unit bypass switch provides hardware-level physical isolation. By connecting the bypass switch in parallel, the signal path (or clock / power) of the format conversion unit can be completely cut off when it is disabled, achieving ultimate static energy saving.

[0035] Optionally, the in-memory computing CIM array is an integer in-memory computing CIM array, whose physical topology requires the input data to be arranged in column-major order; the format conversion unit includes a floating-point to fixed-point conversion module and a data rearrangement / transpose module.

[0036] This design achieves transparent hardware and software adaptation and high-concurrency computing. Through hardware-level data rearrangement, it ensures that the row-first storage data can perfectly match the physical column-first topology of the in-memory computing array, maximizing the parallel computing capabilities of CIM.

[0037] To achieve the above-mentioned objectives, this application provides an artificial intelligence computing system, including the in-memory computing device described in any one of the above descriptions. Attached Figure Description

[0038] Figure 1 A schematic diagram illustrating the steps of a data processing method applied to an artificial intelligence computing system, provided for an embodiment of this application; Figure 2 A schematic diagram illustrating the steps of a data processing method applied to an artificial intelligence computing system, provided for an embodiment of this application; Figure 3 A schematic diagram of the structure of an in-memory computing device provided for an embodiment of this application; Figure 4 A schematic diagram illustrating the execution flow of a neural network inference task provided for an embodiment of this application; Figure 5This is a schematic diagram of the adaptation structure of the in-memory computing array and format conversion unit in the embodiments of this application; Figure 6 This is a time-series data flow diagram of the key-value cache lifecycle in the embodiments of this application; Figure 7 This is a schematic diagram of the weight preprocessing and multiplexing data flow in the pre-filling stage of the embodiments of this application, and a data flow comparison diagram between the traditional scheme and the scheme of this application in the decoding stage. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0040] like Figure 1 As shown, this embodiment provides a data processing method applied to an artificial intelligence computing system, including: Data format conversion steps: Obtain the original data in the first data format, and use the format conversion unit to convert it into a second data format that is compatible with the hardware characteristics of the in-memory computing CIM array 100 of the artificial intelligence computing system; Data synchronization transmission steps: Transmit the data in the second data format synchronously along the first and second paths; Specifically, the second data format data is transmitted to the in-memory computing CIM array 100 via the first path to perform computing tasks; the second data format data is stored as a formatted data copy via the second path so that the artificial intelligence computing system can directly call the formatted data copy without format conversion when it needs the original data again.

[0041] The data processing method provided in this embodiment achieves "one-time data format conversion, multiple data reuses". By synchronously splitting the data after format conversion, the "real-time calculation" and "formatted data copy construction" are completed simultaneously, eliminating the computational and power consumption overhead caused by repeatedly performing preprocessing such as quantization and transposition on the same original data in subsequent inference.

[0042] Optionally, the data processing method provided in this embodiment further includes the following steps: In the data reuse step, when it is detected that the computing task needs to reuse the original data, the formatted data copy corresponding to the original data is directly read and transmitted to the in-storage computing CIM array 100.

[0043] This design achieves a highly simplified and optimized computational pipeline. When the original data needs to be reused, the complex preprocessing steps are skipped by directly reading the formatted copy of the original data, significantly reducing processing latency.

[0044] Optionally, the data reuse step further includes: bypassing / disabling the format conversion unit in response to a computing task requiring the use of the original data again.

[0045] This design reduces the power consumption of the AI ​​computing system by bypassing / disabling the format conversion unit. Furthermore, to further reduce the power consumption of the AI ​​computing system, a low-power mode can be entered when the format conversion unit is bypassed / disabled, reducing its operating voltage or cutting off its power supply.

[0046] Optionally, the data processing method further includes: selectively storing the data in the second data format in the external memory 200 or the in-memory computing CIM array 100, depending on the data size, access frequency and lifespan.

[0047] Optionally, if the original data is key-value cache data of a large language model, a formatted copy of the converted data is stored in external storage 200; the formatted data copy is directly accessed through external storage 200 in subsequent inference rounds or batch inference tasks to support long-lifecycle data reuse.

[0048] If the original data is neural network model weight data or speculative decoding verification feature data, then the formatted data copy after its format conversion is stored in the in-memory computing CIM array 100 to support zero-latency reading through parallel multiply-accumulate operations in the data reuse step.

[0049] In this embodiment, considering the differences in storage space between the external memory 200 and the in-memory computing CIM array 100, the differences in data access frequency, and the differences in the lifecycle of different tasks, large-scale data, low-frequency data, and task data with long lifecycles are preferentially stored in the external memory 200, while small-scale data, high-frequency data, and task data with short lifecycles are preferentially stored in the in-memory computing CIM array 100. Therefore, both storage space utilization and data retrieval latency are taken into account.

[0050] The size, frequency of use, and lifespan of the data can be dynamically defined based on specific application scenarios and hardware conditions; this implementation method does not impose any limitations on these aspects.

[0051] Optionally, the raw data includes key-value cache (KV Cache) data from the large language model; In this scenario, when the large language model generates the current token and calculates the new key-value pair (KV Pair) data, it performs a data format conversion step. Specifically, it executes... The quantization or transpose processing required for key-value pair data to convert the key-value pair data from a first data format to a second data format; In the data synchronization and transmission step, the processed key-value pair data of the second data format is transmitted to the in-memory computing CIM array 100 to perform the computing task, and is also transmitted to the external memory 200 to be stored as a formatted data copy (i.e., key-value pair data of the second data format). In the data reuse step, when generating tokens later, historical key-value cached data can be read by directly calling the formatted data copy without the need for format conversion.

[0052] This design solves the performance bottleneck of long sequence inference. Addressing the significant KV cache conversion pressure in large model inference, it achieves "zero-conversion" fast reading of historical token key-value pairs by storing formatted data copies, greatly improving the speed of long text generation.

[0053] Optionally, the raw data includes large language model weight data. During the prefill phase of the large language model, a data format conversion step is performed, and the model weight data is converted from a first data format to a second data format. After the data format conversion is completed, a data synchronization and transmission step is performed, and the data in the second data format is transmitted to the in-memory computing CIM array 100 to participate in the prefill calculation, and then transmitted to the external memory 200 to be stored as a formatted data copy. During the data decoding stage, a data reuse step is performed, and the in-memory computing CIM array 100 directly reads a formatted copy of the data from the external memory 200 to perform the computing task.

[0054] This design solves the performance bottleneck of long sequence inference. Addressing the significant KV cache conversion pressure in large model inference, it achieves "zero-conversion" fast reading of historical token key-value pair data by storing formatted copies, greatly improving the speed of long text generation.

[0055] Optionally, the in-memory computing (CIM) array is an in-memory computing (CIM) array; the first data format is a floating-point format; the second data format is a fixed-point format data that matches the physical topology of the in-memory computing array, including fixed-point mantissa and shared index.

[0056] This design balances computational accuracy with storage compression. It employs a shared exponent format adapted to the CIM topology, leveraging the high energy efficiency of fixed-point computing while preserving near-floating-point dynamic range and accuracy through exponent sharing.

[0057] like Figure 3 As shown, this embodiment also provides an in-memory computing device, including: The in-memory computing CIM array 100 is used to perform the computational tasks required for neural network inference. The format conversion unit 300 is configured to be connected to the external memory 200 and is used to convert the input data from a first data format to a second data format adapted to the hardware characteristics of the in-memory computing CIM array 100. The data stream splitter controller 400 is connected to the format conversion unit 300, the external memory 200 and the in-memory computing CIM array 100 respectively, and is used to transmit the second data format data (hereinafter also referred to as formatted data) output by the format conversion unit 300 through a first path and a second path respectively, wherein the data is transmitted to the in-memory computing CIM array 100 through the first path and to the external memory 200 through the second path to be stored as a copy of the formatted data; The in-memory computing device may include an external memory interface 500 for connecting the external memory 200 and the format conversion unit 300 to read and write raw data and formatted data copies.

[0058] The instruction controller 600 is connected to the format conversion unit 300, the data stream splitter controller 400 and the external memory interface 500 respectively, and can issue instructions to bypass or disable the format conversion unit 300 when reading formatted data copies later.

[0059] The in-memory computing device provided in this embodiment balances computational accuracy and storage compression ratio. It employs a shared exponent format adapted to the CIM topology, utilizing the high energy efficiency of fixed-point computing power while preserving near-floating-point dynamic range and accuracy through exponent sharing.

[0060] The external storage interface 500 ensures efficient and compatible data exchange. Managing raw data and formatted copies through a unified interface simplifies the addressing and access logic of external storage.

[0061] Optionally, the instruction controller 600 is configured to: control the data stream splitter controller 400 to enable dual-path splitting mode at the initial inference time of the neural network; and control the format conversion unit 300 to enter the off or bypass state at the subsequent formatted data copy reuse time, and directly call the formatted data copy from the external memory 200 and transmit it to the in-memory computing CIM array 100.

[0062] This design enables fully automated energy efficiency management. It can automatically switch between "split mode" and "bypass mode" based on inference time, allowing the hardware to achieve optimal efficiency balance without deep software intervention.

[0063] Specifically, the in-memory computing device also includes a format conversion unit bypass switch 700, which is connected in parallel with the format conversion unit 300 and is capable of bypassing the format conversion unit 300 in response to an instruction from the instruction controller 600.

[0064] The format conversion unit bypass switch 700 can be implemented by a logic gate-controlled selector or by a physical circuit switch; this embodiment does not limit this.

[0065] The format conversion unit bypass switch 700 provides hardware-level physical isolation. By connecting the bypass switch in parallel, the signal path (or clock / power) of the format conversion unit can be completely cut off when it is disabled, achieving ultimate static energy saving.

[0066] The following is combined Figure 4 The following describes the flow of the neural network inference task in conjunction with the data processing method provided in this embodiment: After the neural network inference task is started, at the time of pre-filling or new token generation, the original data in the first data format is obtained. Then, through the format conversion unit, the original data is converted into a second data format adapted to the hardware characteristics of the in-memory computing CIM array. The data in the second data format is split through the first path and the second path. The data is directly transmitted to the in-memory computing CIM array along the first path to execute the current computing task. The data is synchronously written to the external memory along the second path to establish a formatted data copy. During subsequent inference, it is determined whether the data needs to be used again. If it is needed, a formatted copy of the data is read directly from the external storage. While the formatted data copy is being transferred to the in-memory CIM array, the format conversion unit is bypassed / disabled, and the calculation task is executed.

[0067] Optionally, such as Figure 5 As shown, the in-memory computing CIM array 100 is an in-memory computing array. Further, the in-memory computing array is an integer in-memory computing array. The format conversion unit includes a floating-point to fixed-point conversion module and a data rearrangement / transpose module. The second data format is a fixed-point mantissa and a shared exponent that match the physical topology of the in-memory computing array. The in-memory computing device directly loads the fixed-point mantissa and the shared exponent into the storage array of the in-memory computing array through the first path; and stores the fixed-point mantissa and the shared exponent fixed-point data into the external memory through the second path.

[0068] This design achieves transparent hardware and software adaptation and high-concurrency computing. Through hardware-level data rearrangement, it ensures that the row-first storage data can perfectly match the physical column-first topology of the in-memory computing array, maximizing the parallel computing capabilities of CIM.

[0069] Specifically, the format conversion unit 300 can extract the maximum value as a shared exponent by comparing the exponent bits of a set of original data, and then perform shift alignment processing on each fixed-point mantissa based on the shared exponent. In this embodiment, when the format conversion unit 300 converts the original data of the first data format (floating-point format) to the second data format (fixed-point format with shared exponent), the internally integrated exponent processing logic and mantissa alignment logic work together. The specific implementation process is as follows: The format conversion unit 300 first divides the input raw data stream into blocks, forming data blocks consisting of multiple floating-point numbers. The format conversion unit 300 internally includes an exponent parsing circuit to extract the exponent information of each floating-point number in the data block in parallel. Subsequently, all extracted exponent information is input to the comparator tree circuit within the format conversion unit 300. This comparator tree circuit, through multi-level comparison logic, selects the largest exponent value from all exponent values ​​in the data block within a single clock cycle or a preset clock cycle, and determines this as the shared exponent of the data block. This shared exponent serves as the unified reference for subsequent mantissa scaling and is stored along with the converted data.

[0070] The shift calculation and mantissa dynamic alignment are performed as follows: After determining the shared exponent, the format conversion unit 300 uses its internal subtractor array to calculate the numerical difference between the individual exponent value of each floating-point number in the data block and the shared exponent. This difference represents the displacement of each element relative to the reference value. Next, the mantissa portion of each floating-point number is fed into the corresponding barrel shifter. Based on the difference output by the subtractor array, the barrel shifter performs an arithmetic right shift operation on the mantissa portion by the corresponding number of bits. Through this shift operation, the physical weights of all elements within the data block are aligned to a uniform range defined by the shared exponent.

[0071] Implicit bit compensation and fixed-point truncation: During the shifting process, the padding logic of the format conversion unit 300 automatically compensates for the implicit start bit (such as "1") corresponding to the floating-point format in the high-order bits of the mantissa. The aligned high-precision data stream then enters the truncation and saturation logic circuit. This circuit truncates the high-order valid data according to the target fixed-point bit width (such as INT8 or INT4) to form the fixed-point mantissa. At the same time, the saturation logic circuit monitors in real time whether the shifted value exceeds the maximum representation range of the target bit width; if overflow occurs, it automatically clamps the output value to the boundary value of the target bit width (such as the maximum positive value or the minimum negative value) to ensure the stability of the calculation result.

[0072] The format is encapsulated and outputs in two channels. Ultimately, the fixed-point mantissa generated by the above logic and the previously determined shared index together constitute the complete information of the second data format. This formatted data is then sent to the data stream splitter controller 400, thereby enabling direct loading into the in-memory computing CIM array 100 and copy storage into the external memory 200.

[0073] In Computing-in-Memory (CIM) architectures, the design of the physical topology and the arrangement of data flows directly determine computational efficiency and hardware overhead. In integer CIM arrays, the physical topology requires a "column-first" arrangement primarily to match the weight mapping structure and parallel accumulation logic of CIM computation. In-memory computing arrays (such as SRAM or RRAM-based arrays) typically store weights in cross-point cells, with each column corresponding to an output channel. Column-first sorting allows input data to be processed by activating an entire row at once via a row driver, enabling all columns to perform multiply-accumulate (MAC) operations in parallel.

[0074] When data is arranged consecutively in columns, merged memory access can be achieved at the hardware level. For neural network operators (such as convolutional or fully connected layers) that require frequent column-wise summations, column-first approach can significantly reduce the number of address jumps and improve cache hit rate.

[0075] The summation results of in-memory calculations of current or voltage are typically aggregated at the bottom of the bitline via an analog-to-digital converter (ADC) or an adder tree. Column-first sorting ensures that the physical data flow is consistent with the logical summation direction, reducing complex cross-column wiring.

[0076] In this method, the specific implementation of the floating-point to fixed-point conversion module (F2I) is as follows: In-memory computing arrays typically process integer (INT8 / INT4) data to save power, therefore it is necessary to convert the input floating-point numbers (FP32 / FP16) to fixed-point numbers. The implementation steps are as follows: Scaling: First, determine a scaling factor S. The relationship between the fixed-point value Q and the floating-point value V is usually V ≈ S × (Q - Z), where Z is the offset (zero-point).

[0077] The quantization calculation logic is implemented by the hardware circuit through multipliers and shifters to perform the following operation: Q = clamp(round(V / S) + Z) The specific implementation circuit can include a floating-point divider (often replaced by a multiplier multiplied by 1 / S), an adder (to handle zero-point offset), and a saturation cutoff circuit to prevent numerical overflow (such as exceeding the range of -128 to 127 of INT8).

[0078] The core task of the data rearrangement or transpose module is to convert row-major data in external storage (such as DDR or SRAM cache) into the column-major format required by the array.

[0079] The implementation based on transpose buffer is as follows: Dual-port SRAM: A small dedicated SRAM buffer is used. Data is written "rows" but read "columns" by modifying the address logic.

[0080] Implementation logic: Write address Addrin = i × width + j; Read address Addrout = j × height + i.

[0081] Based on linear feedback shift registers (LFSRs) or crossbar switches: For small-scale rearrangements, a cross switch composed of a Multiplexer (MUX) array is used to dynamically change the mapping relationship between input and output according to the control signal, so as to realize the position interchange in a single cycle.

[0082] Pipelining based on Ping-Pong Buffer: To avoid blocking computation, two sets of rearrangement buffers are typically designed. While the first set is performing a "row-to-column" rearrangement and feeding data to the CIM array, the second set is receiving the next batch of raw data after floating-point to fixed-point conversion.

[0083] The format conversion unit 300 may specifically include: The floating-point to fixed-point (F2I) module is used to map values ​​from high-bit-width floating-point (e.g., FP16) to low-bit-width fixed-point (e.g., INT8). It can utilize a scaling factor extraction circuit, employing a comparator to extract the maximum exponent from the data block as the shared exponent. The quantization alignment circuit includes a subtractor (calculating the exponent difference) and a barrel shifter. The mantissa is right-shifted and aligned based on the exponent difference, then a truncation circuit preserves the target bit width. Finally, a saturation detection logic circuit automatically clamps values ​​exceeding the fixed-point representation range to prevent data overflow.

[0084] The data rearrangement and transpose module (Transpose Unit) is used to physically reorganize the "row-first" layout in memory into the "column-first" layout required by the in-memory computing array. It may include: Transpose Buffer: adopting a dual-port SRAM architecture, supporting "row-to-row writing and column-to-column reading", thereby completing matrix transpose in a single cycle; and Crossbar Switch: using a switching network composed of MUX (multiplexer) arrays to dynamically change the physical mapping relationship of data channels according to instructions.

[0085] The pipeline controller is used to achieve seamless connection between data format and physical location conversion. It can include a ping-pong buffer, which uses two sets of buffers to work alternately. One set performs format quantization and transpose, while the other set provides a constant amount of data to the in-memory compute CIM array (CIM array) to ensure that the in-memory compute CIM array does not generate bubbles due to data preprocessing; and a bypass selector, which responds to the instructions of the instruction controller. When a formatted copy is detected in external memory, it directly skips the above-mentioned F2I and transpose logic to achieve data pass-through.

[0086] Optionally, the Data Flow Splitter / Controller 400 is configured to synchronously and losslessly route the converted single data stream to the in-memory computing CIM array (real-time path) and the storage interface (backup path). Its hardware implementation includes: The hardware replication and buffering module, designed to ensure that the two signals do not interfere with each other in terms of load and timing, includes the following: Fan-out Buffer: Utilizing a high-drive-capability buffer array, the single-channel electrical signal output from the format conversion unit is split into two, preventing signal distortion caused by increased load; and Synchronous FIFO (First-In-First-Out): Asynchronous or synchronous FIFOs are set on the first and second paths respectively to compensate for the difference in processing speed between the in-memory computing CIM array (CIM) and external memory (DRAM), and to avoid data loss due to path blockage.

[0087] The data flow splitter controller 400 also includes: a state machine-driven gating logic circuit (FSM-based Steering), which uses a finite state machine (FSM) to dynamically control the data destination according to the current instruction cycle, including: Demultiplexer Logic: Based on signals from the instruction controller, it enables dual outputs during the "initial inference" phase; during the "replica multiplexing" phase, it switches the selection endpoint, bypasses the conversion unit, and locks the memory interface input; and Handshake Coordinator: Monitors the valid signal of the in-memory computing CIM array and the ready signal of the external memory, and releases the split data only when both receivers are ready.

[0088] The data flow splitting controller 400 also includes a bus transaction mapper, which implements logical splitting at the system-on-chip (SoC) level using existing bus protocols; It includes: DMA Mirroring: Configure the Direct Memory Access (DMA) controller to automatically initiate a "shadow write" transaction pointing to the external memory address space in the background while data flows to the input registers of the in-memory computed CIM array; and Arbitration and Routing Unit: In the Network on Chip (NoC), the quantized formatted data packets are simultaneously broadcast to the in-memory computing (CIM) array nodes and the memory controller nodes via routing header information.

[0089] Optionally, the external memory can be DRAM memory or global cache. The external memory includes a raw data area and a formatted data copy area, which are used to store raw data and formatted data copies, respectively. Both are connected to the external memory interface of the in-memory computing device for data reading and writing.

[0090] This embodiment also provides an artificial intelligence computing system, including the in-memory computing device provided in this embodiment.

[0091] By integrating the aforementioned in-memory computing devices, the artificial intelligence computing system can achieve longer battery life, lower response latency, and stronger real-time processing throughput when processing large-scale network models such as Transformers.

[0092] In the inference process of large language models (LLMs), Prefill and Decode are two core stages: The Prefill phase is the first round of computation in which the model processes user input. The model processes all tokens in the Prompt (prompt word) in parallel at once, calculates their embeddings, and generates a complete KV cache.

[0093] The Decode phase is an iterative process in which the model generates responses one by one. The model predicts the next token based on the preceding context (using the stored key-value pairs), and each time a new token is generated, a new round of reasoning is required.

[0094] The following is combined with Figure 6 The core application of KV Cache in the Decode stage of a large language model is explained in detail, and the specific implementation steps are as follows: Data generation time (T_gen): The in-memory computing CIM array (NPU computing unit) completes the calculation of the current token and generates new key-value pairs Knew and Vnew. At this time, Knew and Vnew are in standard FP16 floating-point format (first data format). To perform attention score calculation for the next layer of the network, the key-value pairs Knew and Vnew must be transposed and quantized into an INT8 fixed-point format (second data format) adapted to the in-memory computing array. After the format conversion unit 300 completes the data format conversion, it outputs KV data in INT8 format. Under the control of the instruction controller 600, the data flow splitting controller 400 synchronously splits the data into two paths: the first path is directly sent to the in-memory computing CIM array to perform the multiplication and addition calculation of the current layer's attention score; the second path synchronously writes to the formatted data copy area of ​​the external memory through the external memory interface to establish a formatted KV data copy and complete the storage of historical KV key-value pairs.

[0095] Data reuse time (T_reuse): When generating the next token, attention calculation requires accessing historical key-value data Knew and Vnew. The instruction controller 600 issues an instruction to directly read the converted INT8 formatted key-value data copy from the formatted data copy area of ​​external memory. The data stream is directly transmitted to the in-memory computing array for computation. During this process, the format conversion unit 300 is bypassed / disabled throughout, with no data inflow, completely avoiding repeated transpose and quantization operations, and achieving zero-processing reading of historical key-value data.

[0096] The following is combined with Figure 7 This implementation method is described in the context of model weight preprocessing in the large model prefill stage. The specific implementation steps are as follows: Prefill stage: (1) The inference task is started, and the model weights (first data format) in FP16 floating-point format are read from the external memory and sent to the format conversion unit 300; (2) The format conversion unit 300 converts the FP16 floating-point weights into the fixed-point mantissa + shared exponent format (second data format) of the in-memory computing array. This operation is an inherent step that must be performed in the matrix multiplication and addition calculation in the Prefill stage. (3) The data stream split controller 400 synchronously splits the converted formatted weight data into two paths: the first path is directly sent to the in-memory CIM array to perform context encoding matrix multiplication and addition calculation in the Prefill stage; the second path synchronously writes to the formatted data copy area of ​​the external memory through the external memory interface to establish a formatted weight data copy.

[0097] Decode phase: (1) After entering the Decode stage of Token generation, the model weights of the corresponding layer need to be called in each round of calculation; (2) The instruction controller 600 issues an instruction to directly read the formatted weight data copy that has been converted from the external memory 200 and send it directly into the in-memory computing array to perform multiplication and addition calculations; (3) During this process, the format conversion unit 300 is bypassed / disabled throughout, completely avoiding the redundant overhead of repeatedly performing weighted format conversion in each round of Decode.

[0098] Multi-turn dialogue scenarios using large language models are a core application of large language models. In this scenario, the data processing method provided in this implementation is as follows: In multi-turn dialogue scenarios, each time a user enters a new query, a Prefill phase is triggered. This phase requires reading all historical key-value caches generated from all previous dialogue rounds to complete the context splicing and encoding. In the Decode phase of each dialogue round, the entire historical key-value cache needs to be read again to generate a new token.

[0099] In traditional solutions, the format conversion operation needs to be re-performed on the entire historical KV cache for each round of dialogue during the Prefill phase and for each token generated in each round of Decode, even if the exact same conversion was performed in the previous round, because the external memory only stores the original format data. For example, if a user has 20 rounds of dialogue, the cumulative context length reaches 128K. Traditional solutions would need to perform hundreds of format conversions on the entire historical KV cache, resulting in significant power consumption and latency overhead.

[0100] In multi-turn dialogue scenarios, the solution provided in this embodiment performs a format conversion only once at the generation time for all historical key-value pairs, simultaneously creating a formatted data copy and writing it to external memory. Subsequently, whether in the Decode phase of the current dialogue or in the Prefill and Decode phases of all subsequent dialogues, whenever the historical key-value data needs to be accessed, the formatted data copy is read directly. By bypassing the format conversion unit throughout the process, the redundant preprocessing overhead across rounds and stages is completely eliminated. In actual testing, in a multi-turn customer service dialogue scenario with 20 rounds and a cumulative 128K contexts, the solution of this invention improves overall system energy efficiency by approximately 62% and reduces average single-turn response latency by 51% compared to traditional solutions, completely resolving the performance bottleneck in long-session multi-turn dialogue scenarios.

[0101] To improve hardware utilization, cloud-based large-model inference services typically employ batch inference, merging inference requests from multiple users into a single batch for synchronous execution. In traditional solutions, each prefill and each decode round within a batch requires format conversion operations on the key-value caches of all users within that batch. This preprocessing overhead increases linearly with the batch size, severely limiting the throughput of batch inference.

[0102] In this implementation, each user's KV Cache only performs a format conversion and creates a copy once during generation. Regardless of how many batches it is scheduled for or how many inference calls it is called in the future, the formatted copy is read directly without repeated conversion.

[0103] In multimodal large-scale model inference scenarios, such as in image-text multimodal large-scale models, the image feature KV Cache and text feature KV Cache of the image-text cross-attention module need to be repeatedly read multiple times during inference. In text-to-image diffusion models, the text features output by the text encoder and the intermediate features in the image generation process also need to be repeatedly called in dozens of denoising iterations. All of the above feature data need to undergo format conversion to adapt to the in-memory computing array. In traditional solutions, each call requires repeated preprocessing, which is extremely costly.

[0104] The data processing method of this embodiment performs format conversion only once when image features, text features, and intermediate features are generated for the first time, and a formatted copy is simultaneously created and written to external memory; all subsequent iteration steps and all attention calculation steps directly read the formatted copy, bypassing the format conversion unit throughout the process.

[0105] This implementation method is applied to scenarios that accelerate speculative decoding in large models. The specific implementation method is as follows: Speculative decoding is currently the mainstream acceleration technique for large-scale model inference. It generates draft tokens by a small model and then verifies the correctness of the draft tokens by a large model, which can improve the inference speed by 2-3 times. However, this technique requires reading the entire historical key-value cache multiple times during the verification stage. In traditional solutions, each verification requires format conversion of the key-value cache, which brings extremely high additional overhead and severely limits the actual speedup ratio of speculative decoding.

[0106] In this implementation, the historical KV cache undergoes a format conversion only once during generation. Each verification step in speculative decoding directly reads the formatted data copy, eliminating the need for repeated format conversions and completely removing the additional preprocessing overhead of speculative decoding. Actual testing shows that this implementation increases the actual speedup of speculative decoding from 2.1 times to 3.8 times, while simultaneously improving system energy efficiency by 32%, further amplifying the commercial value of speculative decoding technology.

[0107] The technical solution of the present invention has now been described in conjunction with the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to the specific embodiments described above. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions resulting from such changes or substitutions will all fall within the scope of protection of the present invention.

Claims

1. A data processing method applied to an artificial intelligence computing system, characterized in that, include: Data format conversion steps: Obtain the original data in the first data format, and use the format conversion unit to convert it into a second data format that is compatible with the hardware characteristics of the in-memory computing CIM array of the artificial intelligence computing system; Data synchronization transmission steps: Transmit the data in the second data format synchronously along the first path and the second path respectively; Specifically, the second data format data is transmitted to the in-memory computing CIM array via the first path to perform computing tasks; The second path stores the data in the second data format as a formatted data copy so that the artificial intelligence computing system can directly call upon the original data when it needs it again.

2. The data processing method according to claim 1, characterized in that, Also includes: In the data reuse step, when it is detected that the computing task needs to use the original data again, the formatted data copy corresponding to the original data is read directly.

3. The data processing method according to claim 2, characterized in that, The data reuse step further includes: bypassing / disabling the format conversion unit in response to a computing task requiring the use of the original data again.

4. The data processing method according to any one of claims 1-3, characterized in that, The data processing method further includes: selectively storing the data in the second data format in an external memory or the in-memory computing CIM array according to the data size, access frequency and life cycle.

5. The data processing method according to claim 4, characterized in that, If the original data is key-value cache data of a large language model, then a formatted copy of the converted data is stored in the external memory; the formatted data copy is directly accessed through the external memory in subsequent inference rounds or batch inference tasks to support long-lifecycle data reuse.

6. The data processing method according to claim 4, characterized in that, If the original data is neural network model weight data or speculative decoding verification feature data, then the formatted data copy after its format conversion is stored in the in-memory computing CIM array to support zero-latency reading through parallel multiply-accumulate operations in the data reuse step.

7. The data processing method according to any one of claims 1-3, characterized in that, The raw data includes key-value cache data of the large language model. The data in the first data format is the key-value pair data generated and calculated by the large language model to generate the current token. It is quantized or transposed into formatted key-value data in the second data format through the data format conversion step. The formatted key-value data is transmitted to the in-memory computing CIM array through the first path through the data synchronization transmission step to perform computing tasks. It is stored as a formatted key-value data copy through the second path for direct access by the artificial intelligence computing system.

8. The data processing method according to any one of claims 1-3, characterized in that, The original data includes large language model weight data. During the pre-filling stage of the large language model, the data format conversion step is performed, and the model weight data is converted from a first data format to a second data format. After the data format conversion is completed, the data synchronization and transmission step is performed, and the data in the second data format is transmitted to the in-memory computing CIM array to participate in the pre-filling calculation, and then transmitted to an external memory to be stored as a formatted data copy. During the decoding phase, a data reuse step is performed, in which the in-memory computing CIM array directly reads a formatted copy of data from an external memory to perform computing tasks.

9. The data processing method according to any one of claims 1-3, characterized in that, The data processing method described above is applied to cloud-based batch inference scenarios: For each user request in batch inference, its key-value cache data is only formatted and a formatted copy is created once during generation; When the key-value cached data is subsequently scheduled or used for inference, data reuse is achieved by reading the corresponding formatted data copy.

10. The data processing method according to any one of claims 1-3, characterized in that, The data processing method described above is applied to multimodal large-model inference scenarios: For image feature key value cache data, text feature key value cache data, or intermediate feature data in the diffusion model, data format conversion is performed only once during the first generation, and a formatted data copy is created and written to external storage. In subsequent iterative steps or cross-attention calculation stages, the calculation task is performed by reading the formatted data copy.

11. The data processing method according to any one of claims 1-3, characterized in that, The data processing method is applied to scenarios involving accelerated speculative decoding. Historical key-value cache data undergoes a data format conversion only once during generation; In the verification process of the large model verification draft token, the formatted data copy is read directly to eliminate the repeated format conversion processing of historical key-value cache in the verification process.

12. The data processing method according to any one of claims 1-3, characterized in that, The first data format is a floating-point format; the second data format is a fixed-point format that matches the physical topology of the in-memory computing array, and includes a fixed-point mantissa and a sharing exponent.

13. An in-memory computing device, characterized in that, include: In-memory computing (CIM) arrays are used to perform computational tasks required for neural network inference. The format conversion unit is configured to be connected to an external memory and is used to convert the original data from a first data format to a second data format adapted to the hardware characteristics of the in-memory computing CIM array; A data stream splitter controller is connected to the format conversion unit, the external memory, and the in-memory computing CIM array, respectively, and is used to transmit the data in the second data format output by the format conversion unit via a first path and a second path, wherein the data is transmitted to the in-memory computing CIM array via the first path and to the external memory via the second path to be stored as a formatted data copy. The instruction controller is connected to the in-memory computing CIM array, the format conversion unit, and the data stream splitter controller, respectively. It can respond to the computing request of the in-memory computing CIM array, read the formatted data copy, and issue instructions to bypass or disable the format conversion unit.

14. The in-memory computing device according to claim 13, characterized in that, Also includes: An external storage interface is used to connect the external storage and the format conversion unit to enable the reading and writing of the original data and the formatted data copy.

15. The in-memory computing device according to claim 13, characterized in that, The instruction controller is configured to: at the initial inference time of the neural network, control the data stream splitter controller to enable the data splitting transmission function; when the formatted data copy needs to be called later, control the format conversion unit to enter the off or bypass state, and directly call the formatted data copy from the external memory and transmit it to the in-memory computing CIM array.

16. The in-memory computing device according to claim 13, characterized in that, It also includes a format conversion unit bypass switch, which is connected in parallel with the format conversion unit and connected to the instruction controller, and can bypass the format conversion unit in response to an instruction from the instruction controller.

17. The in-memory computing device according to claim 13, characterized in that, The in-memory computing CIM array is an integer in-memory computing CIM array, and its physical topology requires the input data to be arranged in column-major order; the format conversion unit includes a floating-point to fixed-point conversion module and a data rearrangement / transpose module.

18. An artificial intelligence computing system, characterized in that, The in-memory computing device includes any one of claims 13-17.

Images