A tensor core driven image transposition online conversion device, circuit and method for multi-modal large models

Abstract

The application discloses a kind of tensor kernel driven image conversion online conversion device, circuit and method for multimodal large model, the device is cooperatively designed with tensor processing unit array, adopts multiple channel storage interface and is connected in parallel with multiple GDDR6, input feature map is stored in different GDDR6 according to channel dimension distribution, to realize storage bandwidth aggregation.The device comprises: multiple parallel image conversion conversion modules, each module corresponds a GDDR6 channel, independently executes dynamic data placement based on memory bank perception and sliding window data rearrangement;Inter-channel data interaction network, part of window vector output by each module is spliced into complete matrix multiplication input;And the push circuit directly connected with tensor processing unit.The application is read by multiple channels in parallel, online conversion and calculation flow overlap, eliminates the data inflation and off-chip memory bottleneck of traditional image conversion, can fully utilize the aggregated bandwidth of multiple GDDR6, significantly improves the inference efficiency of multimodal large model on special AI processor.

Description

Technical Field

[0001] This invention relates to the fields of chip design, computational acceleration, and hardware design technology supporting AI inference / training, and particularly to a tensor kernel-driven online image transcoding device, circuit, and method for multimodal large models. Background Technology

[0002] Convolutional Neural Networks (CNNs) and convolution-based Visual Transformers (ViTs) are core components of text-to-image generation models (such as Stable Diffusion, an open-source text-to-image generation model developed by StabilityAI), visual language (Qwen2.5-VL), and multimodal understanding models. In practical hardware deployments, to improve the utilization of computing units, image-to-column (im2col) algorithms are typically used to convert 3D convolution operations into 2D matrix multiplication, thereby utilizing efficient General-Purpose Matrix Multiplication (GEMM) hardware accelerators (such as systolic arrays or tensor processing units) for computation. However, existing technical solutions suffer from the following bottlenecks in practical applications:

[0003] 1. Data bloat and off-chip memory bottleneck: Traditional image transcoding algorithms, when executed in the software layer or offline, require expanding and copying the sliding window data of the input feature map, leading to a dramatic increase in intermediate data volume, which can be tens of times larger under common configurations. This bloated matrix needs to be written back to off-chip DRAM and then read by the GEMM accelerator, resulting in huge off-chip bandwidth pressure and power consumption overhead.

[0004] 2. Fragmented hardware architecture and stagnant pipeline: In existing AI chips, the module that performs image transcoding is usually a direct memory access engine or a dedicated data loading unit, which is often loosely coupled with the tensor processing unit array that performs matrix multiplication. The output of image transcoding needs to be stored in an intermediate buffer first, and then the tensor processing unit retrieves it through an independent data path. This store-and-forward mechanism adds pipeline bubbles and cannot fully utilize the peak computing power of the tensor processing unit.

[0005] 3. Insufficient storage bandwidth utilization: As the number of large model parameters and resolution continue to increase, a single DRAM channel can no longer meet the bandwidth requirements. High-end AI chips generally use multiple GDDR6 / HBM chips, but existing image transcoding designs often only support single-channel reading, failing to fully utilize the aggregated bandwidth of multiple channels, leading to an increasingly prominent memory wall problem.

[0006] 4. Inefficient data arrangement and severe storage conflicts: Traditional feature map storage methods store data continuously by row / channel, causing multiple sampling points to fall on the same storage unit during sliding window reading, resulting in access conflicts, limiting read bandwidth, and making it impossible to acquire all sliding window data in parallel within a single cycle.

[0007] 5. Poor adaptability of multimodal models: With the emergence of large multimodal models such as DeepSeek-OCR and Qwen3, the networks not only contain fixed-size convolutions of traditional CNNs, but also a large number of dynamically shaped inputs, such as variable-length text feature maps in optical character recognition and multi-scale image pyramids in VL models. Existing image transcoding hardware units are usually optimized for fixed sizes, making it difficult to dynamically adjust data flow and placement strategies at runtime, resulting in low actual inference efficiency.

[0008] Therefore, there is an urgent need in the market for an image transcoding solution that can be deeply coupled with tensor processing unit arrays, support online conversion, and dynamically adapt to multimodal model parameters. Summary of the Invention

[0009] This invention aims to solve the following problems in the prior art:

[0010] 1. Eliminate the massive data expansion caused by traditional offline image transcoding and the resulting bottleneck in off-chip DRAM access.

[0011] 2. Achieve seamless pipelined image transcoding and tensor processing unit matrix calculation at the hardware level, avoiding idle operation of tensor processing units caused by data format conversion.

[0012] 3. How to design a multi-channel parallel storage and read architecture to fully utilize the aggregated bandwidth of multiple GDDR6 chips and overcome the storage wall limitation.

[0013] 4. Design a storage-aware architecture that enables all sampling points within any sliding window to be read in parallel within one cycle, eliminating storage conflicts.

[0014] 5. Design a reconfigurable dataflow circuit that is compatible with various computation modes such as standard convolution in Stable Diffusion and other text-based graph models, dilated convolution in Qwen2.5-VL, and deformable convolution in DeepSeek-OCR, and adapts to dynamically changing input feature map sizes in real time.

[0015] To address the above technical problems, this invention proposes a tensor kernel-driven online image transcoding conversion device for multimodal large models. This device, acting as the front-end data engine of a tensor processing unit array, is directly connected to the instruction issuing end and input register file of the tensor processing unit. It employs a multi-channel parallel storage architecture, comprising N independent image transcoding conversion modules, each corresponding to one of N GDDR6 memory channels, and an inter-channel data exchange network. The device also includes N GDDR6 memory channel interface circuits, a direct connection interface circuit for the tensor processing unit, and a dynamic configuration parser.

[0016] Each GDDR6 memory channel interface circuit includes an independent PHY, a controller and a direct memory access engine, supporting high-bandwidth burst read and write with external GDDR6 devices;

[0017] Each image-to-column conversion module independently processes the feature map data of one channel group, and internally includes a multi-bank SRAM array, a dynamic bank mapping circuit, a non-blocking sliding window data rearrangement network, a ping-pong buffer circuit, and a prefetch control circuit;

[0018] The inter-channel data interaction network adopts a multi-stage non-blocking switching network architecture or a crossbar matrix structure, which is used to splice partial window vectors output by N modules into a complete vector, where N≥1;

[0019] The tensor processing unit direct-connect interface circuit includes a ping-pong buffer and a handshake controller, and is directly connected to the input register stack of the tensor processing unit;

[0020] The dynamic configuration parser is configured to receive convolution parameters and storage distribution information from the main control unit, and generate configuration words for each module in real time.

[0021] In the present invention, the C channels of the input feature map are divided into N consecutive channel groups, each group has ceil(C / N) channels, which are respectively stored in different GDDR6; the channel range included in the C_group (0≤C_group<N) channel group is:

[0022] ;

[0023] wherein, ; ceil(·) is the upward rounding for feature map calculation;

[0024] Inside each GDDR6, data is stored continuously in the order of spatial coordinates (h,w), each pixel occupies C_group×D bits, D is the data bit width, and the address mapping formula is:

[0025] ;

[0026] wherein, h is the row index of the input feature map; w is the column index of the input feature map; W is the width of the input feature map; C_group=ceil(C / N), which is the number of channels stored in each GDDR6 channel, N is the number of GDDR6 chips, C is the total number of channels; D is the data bit width; addr is the internal byte address of the memory device.

[0027] In the present invention, the multi-bank SRAM array in the image-to-column conversion module includes B independent single-port SRAM banks, and the bit width of each bank is:

[0028] W_bank = T_c × D bits;

[0029] Where T_c is the number of channels processed in parallel by the tensor processing unit at one time, and D is the data bit width;

[0030] The image transcoding module also includes a storage volume partitioning controller. This controller dynamically calculates address mapping rules based on the convolution parameters. For spatial location (h, w) and channel group c_group, the storage volume index calculation formula is:

[0031] ;

[0032] Where h and w represent the row and column indices of the output feature map; stride_h and stride_w are the stride sizes of the convolution in the height and width directions; offset_h and offset_w are the offsets of the convolution kernel relative to the center point; W is the width of the input feature map; and B is the number of storage volumes.

[0033] In this invention, the image transcoding module further includes an address generation circuit, which adopts a three-stage pipeline structure:

[0034] Level 1, calculating global coordinates:

[0035] ;

[0036] Where (i,j) is the position within the convolution kernel, and dilation is the hole ratio;

[0037] The second level calculates the memory bank index and memory bank address based on global coordinates:

[0038] ;

[0039] ;

[0040] Where W represents the width of the input feature map, B represents the number of storage units, T_c is the number of channels processed in parallel by the tensor processing unit at one time, and c_group is the channel group index;

[0041] The third stage generates read enable and write enable signals to drive the memory array.

[0042] In this invention, the image transcoding module further includes a write data distributor, which contains a routing tree of B channels to distribute streaming data from the direct memory access engine to the corresponding memory bank according to the memory bank index. Each clock cycle can process data from T_c channels.

[0043] The image transcoding module also includes an online conversion circuit coupled to the tensor processing unit. The online conversion circuit includes a window data generator, a ping-pong buffer, a push controller, and a prefetch controller.

[0044] The window data generator contains K_h×K_w parallel address generation units. Each unit independently calculates the coordinates of the sampling point and generates a memory read request. The address generation logic supports dilated convolution and deformable convolution.

[0045] The ping-pong buffer comprises two dual-port SRAM blocks of equal capacity, each with a buffer capacity equal to the data size of a window: K_h×K_w×T_c×D bits, and uses ping-pong operation;

[0046] The push controller is implemented through a state machine to complete the handshake operation with the tensor processing unit.

[0047] The prefetch controller is an independently operating address generator used to pre-calculate the coordinates of the next window, drive the window data generator to prefetch, and store the result in an idle buffer.

[0048] In this invention, the non-blocking sliding window data rearrangement network circuit uses a Benes network or an Omega network to implement multi-level cross switching, which can complete any arrangement within a single cycle;

[0049] The non-blocking sliding window data rearrangement network circuit adopts a multi-stage non-blocking switching network architecture. For N = K_h × K_w inputs, a three-stage multi-stage non-blocking switching network architecture is used.

[0050] Level 1: n1 r1×r1 cross switches;

[0051] Second stage: m n1×n2 cross switches;

[0052] Level 3: n² r²×r² cross switches;

[0053] Where r1 = ceil(N / m), r2 = ceil(N / m), and m is the number of intermediate levels;

[0054] The non-blocking sliding window data rearrangement network circuit also includes a configuration generator, which calculates the control words of each level of switch online according to the target arrangement order using the Benes routing algorithm;

[0055] The data path of the non-blocking sliding window data rearrangement network circuit adopts a fully pipelined structure. After passing through three levels of cross switches, the data is directly output to the ping-pong buffer with a fixed delay of 3 clock cycles.

[0056] The data interaction network between channels adopts a cross-switch matrix structure, which includes N input ports and 1 output port. Each input port can be connected to any position of the output port to realize vector splicing.

[0057] The configuration of the aggregation network is generated by a dynamic configuration parser based on the channel group size and splicing order of each module. The configuration information includes the starting channel index of the output vector of each module, the total length of the output vector, and whether padding is required. The configuration can be written once before running or dynamically updated when switching each convolutional layer.

[0058] In this invention, the device further includes an online data stream generation circuit for dilated / deformable convolution, the online data stream generation circuit including: a dynamic coordinate generator, an interpolation unit circuit, and an offset buffer;

[0059] The dynamic coordinate generator is used for:

[0060] Base coordinate calculation: (base_h, base_w) = (out_h × stride, out_w × stride);

[0061] Dilated convolution: sampling point offset multiplied by the dilation factor: (i×dilation, j×dilation);

[0062] Deformable convolution: Reads (Δx, Δy) from the offset buffer and superimposes it with the base coordinates:

[0063] ;

[0064] ;

[0065] Coordinate quantization: decomposes floating-point coordinates into integer parts (x_int, y_int) and fractional parts (x_frac, y_frac);

[0066] The interpolation unit circuit includes four integer point reading modules, a weight calculation module, and a multiplication accumulation tree, which is implemented using a four-stage pipeline.

[0067] The offset cache is a dedicated SRAM, stored in a similar manner to the feature map, and supports parallel reading with the feature map.

[0068] Based on the above, the present invention also proposes a tensor kernel-driven online image transcoding conversion circuit for multimodal large models. The circuit is a hardware implementation circuit corresponding to the tensor kernel-driven online image transcoding conversion device for multimodal large models. The circuit is designed in conjunction with the tensor processing unit array and includes a multi-channel GDDR6 interface circuit, an N-way parallel image transcoding conversion hardware circuit, an inter-channel data aggregation hardware circuit, and a direct connection interface circuit for the tensor processing unit.

[0069] Based on the above, this invention also proposes an online tensor kernel-driven image transcoding method for multimodal large models. The method is implemented using a tensor kernel-driven image transcoding device for multimodal large models and includes the following steps:

[0070] S1. The dynamic configuration parser receives and parses the convolution parameters and storage distribution information, configures the storage volume mapping rules, address generation mode, and rearranges the network routing table.

[0071] S2. The input feature map is loaded into a multi-bank SRAM array through the direct memory access engine, and the bank data arrangement circuit writes the data into the corresponding bank according to bank_index and bank_addr.

[0072] S3. The prefetch control circuit calculates the coordinates of the first window, the window data generator generates the global coordinates of the sampling points, and reads data in parallel from the multi-memory SRAM.

[0073] S4. Determine the convolution type. If it is deformable convolution, perform bilinear interpolation through the interpolation unit and then send the data into the rearrangement network. If it is standard / dilated convolution, send the data directly into the non-blocking rearrangement network.

[0074] S5. The non-blocking rearrangement network arranges the data in matrix multiplication order and stores it in a ping-pong buffer as the current window data.

[0075] S6. After the tensor processing unit is ready, push the data to the input register of the tensor processing unit, and the tensor processing unit performs matrix multiplication calculation.

[0076] S7. While the tensor processing unit performs the matrix multiplication of the current window, the prefetch control circuit prefetches the data of the next window, switches the ping-pong buffer, and repeats steps S3 to S6 until all output positions are processed.

[0077] The beneficial effects of this invention on the tensor kernel-driven online image transcoding device, circuit, and method for multimodal large models include:

[0078] Maximizing aggregated storage bandwidth: By distributing storage along the channel dimension and using multiple image transpilation modules for parallel reading, the aggregated bandwidth of multiple GDDR6 chips can be fully utilized. In a certain configuration (e.g., four GDDR6-6400 chips), the theoretical aggregated bandwidth can reach over 256GB / s, four times that of a single-channel solution. Actual bandwidth utilization can reach over 92%, effectively alleviating the storage wall problem.

[0079] Significantly reduces off-chip bandwidth pressure: Compared to offline solutions that require writing image transpilation results back to DRAM, this invention uses online transpilation, allowing data to flow directly from the on-chip level-one cache buffer into the tensor processing unit, eliminating the intermediate off-chip write-back and read-back processes. Experimental data shows that for typical networks such as ResNet-50, it can reduce off-chip memory transactions by approximately 50%. The effect is even more significant for generative models like Stable Diffusion that require multiple convolutional iterations.

[0080] Eliminating pipeline bubbles in the tensor processing unit: Through prefetching and ping-pong caching mechanisms, this invention completely hides the image transpilation time behind the computational latency of the tensor processing unit. While the tensor processing unit is performing the Nth matrix multiplication, the hardware front end is already preparing the data required for the N+1th computation, realizing a fully pipelined transformation-computation process and increasing the effective utilization of the tensor processing unit to nearly 100%.

[0081] Eliminating memory conflicts and improving read bandwidth: Dynamic data placement based on memory awareness allows all sampling points within any sliding window to be read in parallel within a single cycle, eliminating memory conflicts inherent in traditional storage methods. Simulation results show that for 3x3 convolutions, read bandwidth can be improved by up to 9 times.

[0082] Dynamic Adaptation to Multimodal Models: Existing technologies typically fix hardware logic for specific convolution sizes. The reconfigurable non-blocking network and dynamic parser of this invention allow switching convolution modes within a single clock cycle, seamlessly supporting standard convolutions in Qwen2.5-VL, deformable convolutions (achieving sub-pixel sampling through interpolation circuits) in DeepSeek-OCR, and transposed convolutions in Stable Diffusion, avoiding the overhead of designing independent hardware modules for different operators.

[0083] Reduce on-chip storage overhead: Due to the use of storage-aware data placement, there is no need to store redundant sliding window unfolded data. Only the original feature map needs to be stored to meet the data rearrangement requirements, and the on-chip storage area can be reduced by about 30% (compared to the solution that requires storing the complete image transformat unfolded matrix).

[0084] Linear scalability: The number of image transcoding modules can be linearly expanded with the number of GDDR6 channels, supporting future storage solutions with higher bandwidth. It has good scalability, and there is no data dependency between modules, making it easy to achieve high parallelism.

[0085] The circuit design of this invention employs a dual verification strategy of FPGA prototyping and ASIC synthesis evaluation to ensure the feasibility of the technical solution and improve the reliability of performance data. The FPGA platform used is the UVHS verification platform, with the FPGA chip model XCVU19P. For ASIC design / process evaluation, the synthesis tool is Synopsys DesignCompiler; power analysis is performed using Synopsys PrimeTime; the placement and routing tool is Cadence Innovus; and the process node is TSMC 14nm. Attached Figure Description

[0086] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0087] Figure 1 This is a block diagram of the overall architecture of the device of the present invention.

[0088] Figure 2 This is a block diagram illustrating the design architecture of the online image transcoding conversion device and circuit of the present invention.

[0089] Figure 3 This is a block diagram of the dynamic data arrangement circuit based on the memory bank according to the present invention.

[0090] Figure 4 This is a block diagram of the non-blocking sliding window data rearrangement network circuit of the present invention.

[0091] Figure 5 This is a block diagram of the interpolation unit circuit of the present invention.

[0092] Figure 6 This is a flowchart of the data processing process of the present invention. Detailed Implementation

[0093] The invention will be further described in detail below with reference to the specific embodiments and accompanying drawings. Except for the contents specifically mentioned below, the processes, conditions, and experimental methods for implementing the invention are all common knowledge and general knowledge in the art, and the invention does not have any particular limitations.

[0094] Figure 1 The figure illustrates the overall connection between the online image transcoding device described in this invention and multiple external GDDR6 memory devices and a tensor processing unit array. The figure includes:

[0095] N GDDR6 storage devices (labeled GDDR6#0, GDDR6#1, ..., GDDR6#N-1) are used to store input feature map data by channel dimension, and each GDDR6 is independently connected to an image transcoding module.

[0096] There are N image transcoding modules (labeled as image transcoding modules #0, #1, ..., #N-1). Each module is connected to the corresponding GDDR6 through an independent storage channel interface. It is responsible for reading the feature map data of a channel group, performing storage-aware data placement and sliding window rearrangement, and outputting a partial window vector.

[0097] The inter-channel data exchange network connects the outputs of all image transcoding modules, concatenating N partial window vectors into a complete matrix multiplication input vector in channel group order.

[0098] The interface circuit connects the output of the aggregation network to the input register file of the tensor processing unit array, and is responsible for handshake control and data push.

[0099] Tensor processing unit arrays perform matrix multiplication operations to complete convolution calculations.

[0100] The arrows in the diagram indicate the direction of data flow, from GDDR6 through the image transcoding module, interactive network, and interface to finally flow into the tensor processing unit; control signals are sent to each module by the dynamic configuration parser to distribute configuration parameters.

[0101] Figure 2 The diagram illustrates the internal circuitry of a single image transcoding module and its connection to an external tensor processing unit array. The diagram includes:

[0102] Dynamic configuration parser (control state machine): Receives convolution parameters (stride, padding, dilation, etc.) and generates configuration words to output to each submodule.

[0103] Data arrangement circuit based on memory bank: It contains memory bank mapping logic and address generator, and is responsible for writing the input feature map data into the multi-memory bank SRAM array in a memory bank-aware manner.

[0104] Multi-bank SRAM array: Contains B independent memory banks for caching channel group data read from the corresponding GDDR6.

[0105] Prefetch control circuit (address generator): An independently operating address generator that pre-calculates the coordinates of the next sliding window and drives data reading.

[0106] Non-blocking sliding window data rearrangement network: rearranges multiple sample point data read from the SRAM array into the column vector order required for matrix multiplication.

[0107] Interpolation unit circuit (dedicated to deformable convolution): performs bilinear interpolation on the floating-point coordinates of deformable convolution, and outputs data with sub-pixel precision.

[0108] Tensor processing unit interface circuit: includes data packing logic, handshake control state machine and ping-pong buffer, directly connected to the input register file of the tensor processing unit array.

[0109] Tensor processing unit array (multiplier + accumulator): performs matrix multiplication operations and completes convolution calculations.

[0110] The diagram illustrates the data path of the three-stage pipeline: "prefetch-rearrange-push".

[0111] Figure 3 The dynamic data arrangement circuit is described in detail, including three parts: the write path, the storage array, and the read path. The diagram includes:

[0112] Write data allocator (B-way split tree): Receives streaming feature map data from the direct memory access engine and distributes the data to the corresponding SRAM based on the storage bank index.

[0113] Multi-bank SRAM array: contains B independent SRAM banks: Bank0, Bank1, Bank2, ..., BankB-1. Each bank has a bit width of T_c × D bits (T_c is the channel parallelism and D is the data bit width).

[0114] Read address arbitrator (memory request distribution + data collection): Receives parallel read requests (in the format (h, w, c_group)) from the window data generator, groups them by memory bank, issues read commands to each memory bank, and collects the returned data vectors.

[0115] Data flow direction: Write data is written from the direct memory access engine to each storage bank via the allocator; read data is output from each storage bank to the subsequent rearrangement network via the arbitrator.

[0116] The diagram uses arrows to indicate the write and read paths, and also labels the parallel requests from the window data generator.

[0117] Figure 4 The diagram illustrates the specific structure of the rearrangement network. The input is a channel vector with 9 sampling points (corresponding to a 3×3 convolution), which is rearranged through a three-stage, multi-level non-blocking switching network architecture before being output to a ping-pong buffer. The diagram includes:

[0118] Input data: Channel vectors of 9 sampling points, labeled as sampling point 0 to sampling point 8, with each vector having a length of T_c.

[0119] Configuration generator: The Benedict routing algorithm is used to calculate the control word of the three-level switch in real time according to the target arrangement order (sampling point index order).

[0120] A three-stage, multi-stage non-blocking switching network architecture consists of three stages: a first-stage crossbar switch, a second-stage crossbar switch, and a third-stage crossbar switch. The number of switches in each stage is dynamically configured based on the number of inputs N=9 and the number of intermediate stages m. The total delay of data passing through the three stages of switches is fixed at 3 clock cycles.

[0121] Output data: The rearranged vectors are arranged in the column order required for matrix multiplication and then fed into the ping-pong buffer.

[0122] The diagram uses arrows to indicate the flow of data from input to output, and labels "ping-pong buffer" at the bottom as the output destination.

[0123] Figure 5 The hardware pipeline architecture of the bilinear interpolation unit dedicated to deformable convolution is described in detail. The figure includes:

[0124] Memory read address generator: Generates four read addresses based on the integer part (x_int, y_int) of the floating-point coordinates (x_real, y_real), corresponding to (x_int, y_int), (x_int+1, y_int), (x_int, y_int+1), and (x_int+1, y_int+1).

[0125] Data is read from four storage banks: V00, V01, V10, and V11, which are channel vectors (each with a length of T_c) corresponding to four integer points.

[0126] Weighted multipliers: Four multipliers multiply V00, V01, V10, and V11 with their corresponding interpolation weights w00, w01, w10, and w11, respectively, to produce four weighted vectors. The weights are calculated from the decimal part of the coordinates.

[0127] Addition tree: Add the four weighted vectors together to obtain the final interpolation result V.

[0128] Output: The interpolation result V (length T_c) is fed into the subsequent rearrangement network.

[0129] The diagram uses pipeline level separators to illustrate the four-stage pipeline structure (address generation, data reading, weight multiplication, and accumulation), and labels the hardware unit corresponding to each operation.

[0130] Figure 6 The complete data processing flow of the device, from initialization to completion of calculations, is illustrated in flowchart form. The main steps include:

[0131] 1. Start → Dynamically Configure the Parser.

[0132] 2. Parse convolution parameters (stride, padding, dilation, etc.) → Configure memory mapping rules, address generation mode, and rearrange network routing tables.

[0133] 3. The Direct Memory Access Engine loads feature maps from multiple storage banks (SRAM) and then writes the data to the storage bank according to bank_index / bank_addr.

[0134] 4. The prefetch control circuit calculates the coordinates of the first window → the window data generator generates the global coordinates of the sampling points.

[0135] 5. Read data in parallel from multiple SRAMs.

[0136] 6. Convolution type determination: If it is deformable convolution, it goes through the interpolation unit (bilinear interpolation); if it is standard convolution or dilated convolution, it is directly fed into the rearrangement network.

[0137] 7. Non-blocking rearrangement network: Arrange data in matrix order → Store in ping-pong buffer (current window data).

[0138] 8. Data storage control waits for the tensor processing unit to be ready → sends the data to the tensor processing unit input register.

[0139] 9. The tensor processing unit performs matrix multiplication calculations.

[0140] 10. Determine if all output positions have been processed: If not, the prefetch control circuit prefetches the data for the next window, switches the ping-pong buffer, and returns to step 4; if yes, then end.

[0141] The diagram uses rectangles to represent processing operations, diamonds to represent decision branches, and arrows to indicate the flow direction, clearly demonstrating the pipeline control logic of "prefetch-conversion-computation" that completely overlaps.

[0142] This invention proposes a tensor kernel-driven online image transcoding device and circuit. The device serves as the front-end data engine of a tensor processing unit array, directly connected to the instruction issuing terminal and input register file of the tensor processing unit. To achieve efficient acceleration of various convolution operators in multimodal large models, including Stable Diffusion, Qwen2.5-VL, and DeepSeek-OCR, the device adopts a multi-channel parallel storage architecture, comprising N independent image transcoding modules, each corresponding to N GDDR6 memory channels, and an inter-channel data exchange network. The following details the overall architecture, hardware implementation of each core module, and data flow.

[0143] I. Overall Architecture and Module Division

[0144] Figure 1is an overall architecture block diagram of the apparatus. The apparatus includes the following main modules:

[0145] 1. N GDDR6 memory channel interface circuits: Each interface includes an independent PHY, a controller and a direct memory access engine, supporting high-bandwidth burst read and write with external GDDR6 devices.

[0146] 2. N image-to-column conversion modules: Each module independently processes the feature map data of one channel group, and internally includes: a multi-bank SRAM array (B banks); a dynamic bank mapping circuit; a non-blocking sliding window data rearrangement network; a ping-pong buffer circuit; and a prefetch control circuit (address generation).

[0147] 3. Inter-channel data interaction network: Adopting a multi-stage non-blocking switching network architecture or a crossbar matrix structure, the partial window vectors output by the N modules are spliced into a complete vector.

[0148] 4. Tensor processing unit direct interface circuit: Including a ping-pong buffer and a handshake controller, which is directly connected to the input register stack of the tensor processing unit.

[0149] All modules are uniformly controlled by a dynamic configuration parser, which receives convolution parameters and storage distribution information from the main control unit, and generates configuration words for each module in real time. The convolution parameters include stride, dilation rate, convolution kernel size, padding, etc.

[0150] II. Multi-Channel Memory and Data Arrangement

[0151] Storage distribution strategy: The C channels of the input feature map are divided into N consecutive channel groups, each group has ceil(C / N) channels, which are respectively stored in different GDDRs. Let the channel range included in the C_group (0≤C_group<N) channel group be , where . Inside each GDDR6, data is stored consecutively in the order of spatial coordinates (h,w), each pixel occupies bits, D is the data bit width, for example, FP16 is 16 bits. The address mapping formula is:

[0152] .

[0153] where h is the row index of the input feature map, with a value of 0 ≤ h < H, and H is the height of the input feature map, preferably 64-1024; w is the column index of the input feature map, with a value of 0 ≤ w < W, and W is the width of the input feature map, preferably 64-1024; W ranges from 64 to 2048, depending on the input resolution of the model; C_group=ceil(C / N), which is the number of channels stored in each GDDR6 channel, preferably 1-12, N is the number of GDDR6 chips, and C is the total number of channels; D is the data bit width, which is 8 (INT8), 16 (FP16 / BF16), 32 (FP32); addr is the internal byte address of the memory device, which is limited by the capacity of the memory particle.

[0154] When the tensor processing unit needs to calculate a certain output position, N image-to-column modules simultaneously read C_group channels of data at the same spatial position from their corresponding GDDR6 channels. Since each channel group is stored independently, N read requests can be executed in parallel to make full use of the multi-channel aggregated bandwidth.

[0155] III. Composition of the image-to-column conversion module

[0156] 1. Dynamic data permutation circuit based on bank awareness. The storage manner of feature map data in the on-chip SRAM directly affects the parallelism of sliding window reading. Traditional continuous storage will cause multiple points in the same sliding window to fall into the same memory bank, resulting in access conflicts. The bank-aware permutation of the present invention enables all sampling points in any sliding window to be read out from different memory banks at the same time.

[0157] Multi-bank SRAM array: comprising B independent single-port SRAM banks, each bank has a bit width of W_bank=T_c×D bits, where T_c is the number of channels (e.g., 16 channels) processed in parallel by the tensor processing unit at a time, and D is the data bit width (16 bits for FP16)

[0158] The number B of memory banks is the number of banks of the SRAM array inside each image-to-column module, and its selection needs to meet the following conditions:

[0159] 1. Conflict-free parallel reading: For a convolution kernel with a height of K_h and a width of K_w, it shall be ensured that all sampling points in any sliding window can be mapped to different memory banks. Since the address mapping adopts a modulo operation, when B ≥ K_h × K_w, the conflict probability is extremely low. A typical 3×3 convolution requires B ≥ 9.

[0160] 2. Hardware complexity: The larger B is, the more the number of memory banks, but the depth of each memory bank decreases, and the complexity of address decoding and arbitration logic increases. The value of B is preferably an integer power of 2 to simplify the modulo operation.

[0161] 3. Matching the parallelism of the tensor processing unit channels: The bit width of each memory bank is T_c×D, and B and T_c together determine the total bandwidth of each module.

[0162] For a 3×3 convolution, B=16 is sufficient to ensure no collisions, with a probability >99% that the 9 points are distributed across 16 memory locations. For 5×5 or 7×7 convolutions, B=32 is preferred. B=16 is a compromise, with moderate hardware overhead.

[0163] The storage capacity of each storage unit depends on the size of the input feature map, the number of channels handled by the module, and the number of storage units.

[0164] Let the input feature map size be H×W; the number of channels handled by the module be C_group; the number of channels processed by the tensor processing unit at one time be T_c (e.g., 16); and the number of storage units be B.

[0165] The number of channel slices that each memory bank needs to store is: num_slices = ceil(H × W × (C_group / T_c) / B); the size of each channel slice is T_c × D bits, so the storage capacity of each memory bank is: bank_capacity = num_slices × T_c × D (bits).

[0166] The typical capacity of a single-port SRAM memory bank is 64KB. For larger feature maps such as 256×256, it can be increased to 256KB.

[0167] Minimum size: 16KB for low-resolution models; Typical size: 64KB-256KB for most multimodal models; Maximum size: 1MB for high-resolution inputs, such as 1024×1024.

[0168] Storage partitioning controller: Dynamically calculates address mapping rules based on convolution parameters. For spatial location (h, w) and channel group c_group, the storage index calculation formula is:

[0169]

[0170] Where h and w represent the row and column indices of the output feature map, i.e., the currently calculated output pixel positions; stride_h / stride_w is the convolution stride (the sliding step size in the height and width directions), and offset_h / offset_w is the offset within the convolution kernel relative to the center point. For example, for a 3x3 convolution, if the kernel position is (i,j), where... (Considering padding), then offset_h = i, offset_w = j. W is the width (number of columns) of the input feature map. This is the row span (stride) when the feature map is stored row by row in memory, usually equal to the width of the feature map. B is the number of memory banks. The on-chip SRAM is divided into B independent memory banks, each of which can be read and written independently.

[0171] This formula calculates the global linear address of the current sampling point in the input feature map. Then, the linear address is modulo B to determine which memory bank the pixel data should be stored in. Since the global coordinates of different sampling points are different, the probability of the data falling into different memory banks after modulo is very high, thus ensuring that there are no conflicts during parallel reading.

[0172] The address generation circuit adopts a three-stage pipeline structure:

[0173] Level 1: Calculate global coordinates:

[0174] ;

[0175] Where (i,j) represents the position within the convolution kernel.

[0176] Level 2: Calculate the memory index and memory address based on global coordinates:

[0177]

[0178]

[0179] Where W represents the width of the input feature map. B represents the number of storage blocks. In hardware description languages, integer division (rounding down) usually corresponds to truncated division, i.e., discarding the remainder and keeping only the quotient. T_c is the number of channels processed in parallel by the tensor processing unit at one time (e.g., 16). Each storage unit in each storage block stores data from T_c channels, called a channel slice. c_group is the channel group index. Since the total number of channels C may be greater than T_c, the channels need to be divided into multiple groups, each group corresponding to a channel slice. The value of c_group ranges from 0 to ceil(C / T_c)-1.

[0180] This formula calculates the internal address of the sampled data within a specified memory bank. First, the global linear address `global_h × W + global_w` is calculated. Since the number of memory banks is B, pixels with the same linear address are stored sequentially in a round-robin fashion: pixels with linear addresses 0, 1, ..., B-1 are placed in bank0, bank1, ..., bankB-1 respectively; pixels with linear addresses B, B+1, ..., 2B-1 are placed in bank0, bank1, ... . Therefore, the linear addresses of pixels belonging to the same memory bank satisfy `linear_addr % B = bank_index`, and their storage order within the memory bank is determined by... The decision is made (i.e., which pixel is in the memory cell). Then, it's multiplied by T_c because each memory cell address stores data for T_c channels, so the pixel's storage location must be aligned according to the channel slice size. Finally, c_group is added because different channel groups for the same pixel need to be stored at different addresses within the same memory cell.

[0181] Level 3: Generate read enable and write enable signals to drive the memory array.

[0182] Write data allocator: A routing tree containing B channels that distributes streaming data from the direct memory access engine to the corresponding memory bank according to the memory bank index. It can process T_c channels of data per clock cycle.

[0183] 2. Online conversion circuit coupled to the tensor processing unit

[0184] The image transcoding hardware module is directly used as the front-end data supply unit for the tensor processing unit. The converted data is directly pushed into the input register file of the tensor processing unit through a dedicated data path. By utilizing ping-pong buffering and data prefetching mechanisms, the image transcoding transformation and tensor processing unit computation are fully piped.

[0185] Window data generator: Contains 9 parallel address generation units (3x3 sliding windows), each unit independently calculates the coordinates of the sampling points and generates a memory read request. The address generation logic supports dilated convolution (multiplied by the dilation factor) and deformable convolution (stacked offsets).

[0186] Ping-Pong Buffer: Two equal-capacity dual-port SRAM blocks, each with a buffer capacity equal to the data size of one window: K_h × K_w × T_c × D bits. Ping-pong operation is used: Bank A stores the data currently being pushed to the tensor processing unit; Bank B stores the data to be prefetched for the next window.

[0187] Push controller: Implemented through a state machine, it completes the handshake operation with the tensor processing unit.

[0188] Prefetch Controller: An independently operating address generator that pre-calculates the coordinates of the next window, drives the window data generator to prefetch, and stores the results in an idle buffer.

[0189] 3. Non-blocking sliding window data rearrangement network circuit

[0190] The raw data read from the storage array (continuously grouped by channel) needs to be rearranged into the column vector order required for matrix multiplication. Multi-level cross switches can be implemented using Benesian or Omega networks, allowing for arbitrary arrangement within a single cycle.

[0191] Network topology: A multi-level non-blocking switching network architecture is adopted. For N = K_h × K_w inputs (e.g., 9 sampling points), a three-level multi-level non-blocking switching network architecture is used.

[0192] Level 1: n1 r1×r1 cross switches

[0193] Second level: m n1×n2 cross switches

[0194] Level 3: n² r²×r² cross switches

[0195] Where r1=ceil(N / m), r2=ceil(N / m), and m is the number of intermediate levels.

[0196] The configuration generator calculates the control words for each level of switch online using the Benesian routing algorithm, based on the target arrangement order (sampling point index order). For deformable convolutions, the configuration is determined by the sampling order given by the dynamic coordinate generator.

[0197] The data path adopts a fully pipelined structure. After passing through three levels of cross switches, the data is directly output to the ping-pong buffer with a fixed delay of 3 clock cycles.

[0198] IV. Inter-channel data exchange network

[0199] The network structure and function are as follows: N image transcoding modules each output a partial window vector, with each vector having a length of K_h×K_w×T_c. Since each module processes different channel groups, these vectors are continuous in the channel dimension. The task of the inter-channel aggregation network is to concatenate the N partial vectors into a complete vector with a length of K_h×K_w×T_c×N (when T_c is the same for all modules).

[0200] The network employs a crossbar switch matrix structure, containing N input ports and 1 output port. Each input port can be connected to any position on the output port, enabling vector concatenation.

[0201] The configuration of the aggregation network is generated by a dynamic configuration parser based on the channel group size and concatenation order of each module. Configuration information includes: the starting channel index of each module's output vector; the total length of the output vector; and whether padding is needed (when the last group has fewer than T_c channels). The configuration can be written once before runtime or updated dynamically during each convolutional layer switch.

[0202] While the tensor processing unit computes the current window, the aggregation network simultaneously reads partial vectors of the next window from the ping-pong buffers of each module, concatenates them, and stores them in its own small buffer or pushes them directly to the push controller. This process is pipelined in parallel with the prefetching operations of each module.

[0203] V. Online Data Stream Generation Circuit for Drilled / Deformable Convolution

[0204] Dilated convolution is implemented by modifying the address generation logic, while deformable convolution requires dynamic sampling point coordinates and bilinear interpolation. The dynamic coordinate generator of this invention calculates the actual sampling coordinates in real time based on the dilation rate or offset, and drives the interpolation unit to obtain sub-pixel precision values ​​from the feature map.

[0205] Dynamic coordinate generator:

[0206] Basic coordinate calculation:

[0207] Dilated convolution: Sampling point offset multiplied by the dilation factor.

[0208] Deformable convolution: Reads (Δx, Δy) from the offset buffer and superimposes it with the base coordinates:

[0209]

[0210]

[0211] Coordinate quantization: Decomposes floating-point coordinates into integer parts (x_int, y_int) and fractional parts (x_frac, y_frac).

[0212] The interpolation unit circuit (deformable convolution only) includes: reading four integer points, i.e., generating four read requests based on (x_int, y_int): (x_int, y_int), (x_int+1, y_int), (x_int, y_int+1), (x_int+1, y_int+1); weight calculation: w00=(1-x_frac)×(1-y_frac), w01=x_frac×(1-y_frac), w10=(1-x_frac)×y_frac, w11=x_frac×y_frac; multiplication and accumulation tree: the channel vectors of the four integer points are multiplied by their corresponding weights, and then added together to obtain the interpolation result. A four-stage pipeline is used (stage 1: read four integer point data; stage 2: calculate weights using a multiplier; stage 3: perform dot multiplication (T_c parallel operations); stage 4: perform accumulation using an addition tree).

[0213] Offset cache: Dedicated SRAM, stored in a similar manner to the feature map, and supports parallel reading with the feature map.

[0214] The offset buffer stores the sample point offsets required for deformable convolutions (DCNs). Each output location corresponds to K = K_h × K_w sample points, and each sample point contains two offsets (Δx, Δy), typically represented by 16-bit signed integers (precision 1 / 16 pixel or higher). The offset map size is the same as the input feature map: H × W × 2K. Each offset occupies D_offset bits (typically 16 bits).

[0215] Therefore, the total storage capacity required for the offset buffer is: offset_buffer_capacity = H × W × 2K × (D_offset / 8) bytes;

[0216] Value range:

[0217] Small model (32×32): ≤64KB;

[0218] Medium model (256×256): ≤4MB;

[0219] Large model (1024×1024): ≤64MB (in this case, block processing is usually used, and the cache only stores the current block);

[0220] In practical hardware design, to save on-chip SRAM, the offset cache can be configured as a row-by-row cache, or only cache the offsets required by the currently being processed window. Preferred capacity: 128KB ~ 512KB, sufficient to cover a single convolutional layer of most optical character recognition and vision models.

[0221] To support parallel reading of the feature map, the offset cache should adopt the same memory bank organization structure as the feature map's multi-bank SRAM array, i.e., divided into B memory banks, each with the same bit width (T_c×D), and the address mapping rules should also be consistent. This ensures that the offset can be read in parallel using the same coordinate index while reading the feature map, without the need for additional address translation. The preferred number of memory banks is: consistent with the feature map SRAM, B=16 or B=32.

[0222] Example 1: U-Net convolution acceleration for Stable Diffusion text-based graph models

[0223] To verify the acceleration effect of the present invention on standard convolutional layers in the context of large-scale graph inference, the focus is on evaluating the degree to which the online conversion mechanism alleviates off-chip memory access pressure and improves the utilization rate of tensor processing units.

[0224] The core denoising network U-Net of latent diffusion models (LDMs) such as Stable Diffusion contains a large number of convolutional layers. In practical deployments, these convolutional layers face significant engineering challenges. According to technical analysis of diffusion models, when the image resolution increases from 64×64 to 512×512, the spatial size increases by 64 times, and the computational cost of convolution increases by approximately 64 times accordingly. More importantly, diffusion models require 20-50 iterative sampling steps, leading to the need for multiple repetitions of the expensive forward propagation. Traditional offline image transcoding schemes unfold the feature map before each convolution, generating a large amount of intermediate data, causing off-chip DRAM bandwidth to become a system bottleneck.

[0225] The circuit configuration of this invention:

[0226] Tensor processing unit size: 16×16 systolic array, supporting INT8 computation; Number of memory banks B: 16 independent SRAM banks; Channel parallelism T_c: 16 channels / cycle; On-chip feature map cache: 512KB (multi-bank SRAM array); Ping-pong buffer: 2×4KB dual-port SRAM.

[0227] Baseline Scheme 1 (Offline Image Transcoding): The software performs image transcoding, expands the matrix, stores it back into DRAM, and then reads it by the tensor processing unit.

[0228] Baseline Scheme 2 (Existing Real-Time Processing Scheme): Employs multiple processing units + large-capacity intermediate buffer. Reference paper: Jianghui Wu, Jianpeng Fan, An energy-efficient data-reused real-time image transcoding for OS-dataflow systolic array-based accelerator. Microelectronics Journal. Volume 165, November 2025, 106831.

[0229] The test data is shown in the table below.

[0230]

[0231] The test data shows that, firstly, off-chip memory access is significantly reduced. Compared to offline image transcoding, the proposed solution reduces DRAM access by 88.5%. Traditional offline solutions require writing the expanded matrix (9×512×64×64≈18.9MB) back to DRAM for the tensor processing unit to read. The proposed solution, through online transcoding, only requires loading the original feature map (64×64×512≈2.1MB) from DRAM, eliminating the need for intermediate data write-back and rereading. This result is largely consistent with the conclusion of the latest research on the LMAI2C method, which reduces DDR access by 79.8%.

[0232] Second, the intermediate buffer is significantly reduced. Existing real-time processing solutions require a 36KB intermediate buffer to store temporary data for multiple sliding windows. This invention, through storage volume-aware placement and ping-pong prefetching, compresses the buffer to 4KB, requiring only the data of one window, a reduction of 88.9%.

[0233] The utilization rate of the third tensor processing unit was increased to 97.2%. In offline solutions, due to the sequential execution of data preparation and computation, the tensor processing unit experiences long idle waiting times. This invention achieves complete overlap between prefetching and computation, minimizing pipeline bottlenecks. The measured utilization rate of 97.2% is close to the theoretical limit.

[0234] Fourth, the hardware overhead is extremely low. The image transcoding module consumes only 610 LUTs and 6.51mW of power. Compared to existing solutions, resource consumption is halved, making it more suitable for resource-constrained edge deployments.

[0235] In the standard convolutional scenario of Stable Diffusion-like image models, this invention successfully eliminates the data bloat and off-chip memory access bottlenecks caused by traditional image transcoding through online transformation coupled with tensor processing units and dynamic data placement based on memory volume awareness. This increases the utilization rate of tensor processing units to over 97%, while minimizing hardware overhead. For diffusion model inference requiring 20-50 iterations, the overall end-to-end latency can be reduced by approximately 50%, demonstrating significant engineering practical value.

[0236] Example 2: Deformable Convolution of DeepSeek-OCR Text Recognition Model

[0237] To verify the hardware acceleration effect of this invention on deformable convolution in optical character recognition tasks, the focus is on evaluating the performance of interpolation unit pipeline design and online data stream generation.

[0238] Text recognition models such as DeepSeek-OCR need to handle curved and slanted text lines, and the fixed sampling grid of standard convolution is insufficient to cover irregular areas. Deformable convolution adapts to geometric deformation by learning the offset of sampling points, but it introduces additional computational overhead: coordinate mapping, bilinear interpolation, and data collection. Existing solutions typically handle these operations on CPUs or dedicated DSPs, resulting in frequent data transfer between the CPU and accelerator, becoming a performance bottleneck.

[0239] The circuit configuration of this invention is as follows:

[0240] Interpolation unit: Four-stage pipeline, integer point reading, weight calculation, dot multiplication, accumulation, supports FP16 calculation; Offset cache: Dedicated 128KB SRAM, arranged according to memory volume awareness; Dynamic coordinate generator: Supports floating-point coordinate calculation with a precision of 1 / 16 pixel.

[0241] The test model and layer configuration selected were deformable convolutional layers in the DeepSeek-OCR backbone network: input feature map 32×32×256; convolution kernel 3×3 deformable convolution, stride 1; offset map 32×32×18 (9 sampling points × 2 coordinates); batch size 1 (single optical character recognition image).

[0242] Baseline Scheme 1 (CPU Processing): Coordinate calculations and interpolation are performed by the ARM Cortex-A53, and data is moved through the Direct Memory Access Engine.

[0243] Baseline Solution 2 (Direct Memory Access Engine Acceleration): The CPU calculates coordinates, the Direct Memory Access Engine moves data, and the hardware performs matrix multiplication.

[0244] Baseline 3 (no interpolation hardware): Only supports standard convolutions; deformable convolutions fall back to the CPU.

[0245] The test data is shown in the table below.

[0246]

[0247] The test data shows that the first inference time is accelerated by 4.6 times. This invention compresses the single-layer inference time from 285.6 μs to 62.4 μs, which is better than the 176.3 μs acceleration of the direct memory access engine. This improvement is consistent with the trend of 69 times speedup (relative to CPU) reported by the LMAI2C method. The difference stems from the fact that LMAI2C compares to a pure CPU baseline, while this embodiment compares to an optimized direct memory access engine acceleration baseline.

[0248] The second data transfer volume is reduced by 98.4%. In traditional solutions, after the CPU calculates the coordinates, the offset and feature map need to be transferred to DDR and then read by the accelerator. This invention integrates the offset cache on-chip, allowing the dynamic coordinate generator to read it directly, and the interpolation result to be directly sent to the tensor processing unit, eliminating most of the data movement between on-chip and off-chip. The measured data transfer volume is reduced from 1152KB to 18KB, requiring only the loading of the original feature map and offset.

[0249] The third interpolation unit implements a 4-cycle pipeline. Bilinear interpolation at each sampling point requires reading 4 integer points, calculating weights, and performing multiplication and accumulation. This invention employs a four-stage pipeline design, completing the interpolation of one sampling point per cycle with a fixed latency of 4 cycles, which is 11 times faster than the CPU's 48 cycles. This design completely hides the interpolation operation within the data preparation pipeline, without affecting the main computation path.

[0250] The utilization rate of the fourth tensor processing unit jumps to 94.5%, while the utilization rate of the tensor processing unit in the traditional scheme is only 21.3% due to the serial data preparation and calculation. This invention generates online data streams, making the calculation mode of deformable convolution no different from that of standard convolution, allowing the tensor processing unit to operate at full capacity.

[0251] In the context of deformable convolution in DeepSeek-OCR, this invention integrates an interpolation unit and a dynamic coordinate generator to hardware-enable and pipeline complex operations that originally required CPU processing, achieving an end-to-end speedup of 4.6 times. This design enables deformable convolution to operate as efficiently as standard convolution on the tensor processing unit, demonstrating significant practical value for optical character recognition models that process curved text.

[0252] Unless otherwise defined, all technical and scientific terms used in this invention have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0253] As used in this invention, the term "comprising" is an open-ended expression, meaning it includes the contents specified in this invention but does not exclude other aspects.

[0254] As used in this invention, the term "and / or" includes any one or more of the related listed items and all combinations thereof.

[0255] The scope of protection of this invention is not limited to the above embodiments. All variations and advantages that can be conceived by those skilled in the art within the technical scope disclosed in this invention are included in this invention and are protected by the appended claims.

Claims

1. A tensor kernel-driven online image transcoding device for multimodal large models, characterized in that, Said device, as a front-end data engine of a tensor processing unit array, is directly connected to the instruction transmitting end and the input register file of the tensor processing unit, adopts a multi-channel memory parallel architecture, and includes N independent image-to-column conversion modules respectively corresponding to N GDDR6 memory channels, as well as an inter-channel data interaction network; said device further includes N GDDR6 memory channel interface circuits, a tensor processing unit direct connection interface circuit, and a dynamic configuration parser; among them, Each GDDR6 memory channel interface circuit includes an independent PHY, a controller and a direct memory access engine, and supports high-bandwidth burst read and write with external GDDR6 devices; Each image-to-column conversion module independently processes the feature map data of one channel group, and internally includes a multi-bank SRAM array, a dynamic bank mapping circuit, a non-blocking sliding window data rearrangement network, a ping-pong buffer circuit, and a prefetch control circuit; The multi-bank SRAM array in said image-to-column conversion module includes B independent single-port SRAM banks, and the bit width of each bank is: W_bank=T_c×D bits; Wherein, T_c is the number of channels processed in parallel by the tensor processing unit at one time, and D is the data bit width; Said image-to-column conversion module further includes a bank division controller, said bank division controller dynamically calculates an address mapping rule according to convolution parameters, and for the spatial position (h,w) and the channel group c_group, the calculation formula of the bank index is: ； Wherein, h and w represent the row and column indices of the output feature map; stride_h and stride_w are the sliding strides of the convolution in the height and width directions, offset_h and offset_w are the offsets relative to the center point in the convolution kernel, W is the width of the input feature map, and B is the number of banks; Said inter-channel data interaction network adopts a multi-stage non-blocking switching network architecture or a crossbar matrix structure, and is configured to splice partial window vectors output by the N modules into a complete vector; Said tensor processing unit direct connection interface circuit includes a ping-pong buffer and a handshake controller, and is directly connected to the input register stack of the tensor processing unit; Said dynamic configuration parser is configured to receive convolution parameters and storage distribution information from the main control unit, and generate configuration words for each module in real time.

2. The tensor kernel-driven online image transcoding device for multimodal large models according to claim 1, characterized in that, The C channels of the input feature map are divided into N consecutive channel groups, each group has ceil(C / N) channels, which are respectively stored in different GDDR6s, and ceil(·) is the upward rounding in feature map calculation; for the C_group-th channel group where 0≤C_group<N, the included channel range is: ； in, ; Inside each GDDR6, data is stored continuously in the order of spatial coordinates (h,w), each pixel occupies C_group×D bits, D is the data bit width, and the address mapping formula is: ； Where h is the row index of the input feature map; w is the column index of the input feature map; W is the width of the input feature map; C_group=ceil(C / N) is the number of channels stored for each GDDR6 channel, N is the number of GDDR6 chips, and C is the total number of channels; D is the data bit width; addr is the storage device, internal byte address.

3. The tensor kernel-driven online image transcoding device for multimodal large models according to claim 1, characterized in that, The image transcoding module also includes an address generation circuit, which adopts a three-stage pipeline structure. Level 1, calculating global coordinates: ； Where (i,j) is the position within the convolution kernel, and dilation is the hole ratio; The second level calculates the memory bank index and memory bank address based on global coordinates: ； ； Where W represents the width of the input feature map, B represents the number of storage units, T_c is the number of channels processed in parallel by the tensor processing unit at one time, and c_group is the channel group index; The third stage generates read enable and write enable signals to drive the memory array.

4. The tensor kernel-driven online image transcoding device for multimodal large models according to claim 1, characterized in that, The image transcoding module also includes a write data distributor, which contains a routing tree of B channels to distribute streaming data from the direct memory access engine to the corresponding memory bank according to the memory bank index, and processes data from T_c channels per clock cycle. And / or, The image transcoding module also includes an online conversion circuit coupled to the tensor processing unit. The online conversion circuit includes a window data generator, a ping-pong buffer, a push controller, and a prefetch controller. The window data generator contains K_h×K_w parallel address generation units. Each unit independently calculates the coordinates of the sampling point and generates a memory read request. The address generation logic supports dilated convolution and deformable convolution. The ping-pong buffer comprises two dual-port SRAM blocks of equal capacity, each with a buffer capacity equal to the data size of a window: K_h×K_w×T_c×D bits, and uses ping-pong operation; The push controller is implemented through a state machine to complete the handshake operation with the tensor processing unit. The prefetch controller is an independently operating address generator used to pre-calculate the coordinates of the next window, drive the window data generator to prefetch, and store the result in an idle buffer.

5. The tensor kernel-driven online image transcoding device for multimodal large models according to claim 1, characterized in that, The non-blocking sliding window data rearrangement network circuit uses a Benesian network or an Omega network to implement multi-level cross switching, which can complete any arrangement in a single cycle; The non-blocking sliding window data rearrangement network circuit adopts a multi-stage non-blocking switching network architecture. For N = K_h × K_w inputs, a three-stage multi-stage non-blocking switching network architecture is used. Level 1: n1 r1×r1 cross switches; Second stage: m n1×n2 cross switches; Level 3: n² r²×r² cross switches; Where r1 = ceil(N / m), r2 = ceil(N / m), and m is the number of intermediate levels; The non-blocking sliding window data rearrangement network circuit also includes a configuration generator, which calculates the control words of each level of switch online according to the target arrangement order using the Benes routing algorithm; The data path of the non-blocking sliding window data rearrangement network circuit adopts a fully pipelined structure. After passing through three levels of cross switches, the data is directly output to the ping-pong buffer with a fixed delay of 3 clock cycles.

6. The tensor kernel-driven online image transcoding device for multimodal large models according to claim 1, characterized in that, The data interaction network between channels adopts a cross-switch matrix structure, which includes N input ports and 1 output port. Each input port can be connected to any position of the output port to realize vector splicing. The configuration of the aggregation network is generated by a dynamic configuration parser based on the channel group size and concatenation order of each module. The configuration information includes the starting channel index of the output vector of each module, the total length of the output vector, and whether padding is required. The configuration is written once before running or dynamically updated when switching each convolutional layer.

7. The tensor kernel-driven online image transcoding device for multimodal large models according to claim 1, characterized in that, The device also includes an online data stream generation circuit for dilated / deformable convolution, the online data stream generation circuit including: a dynamic coordinate generator, an interpolation unit circuit and an offset buffer; The dynamic coordinate generator is used for: Base coordinate calculation: (base_h, base_w) = (out_h × stride, out_w × stride); Dilated convolution: sampling point offset multiplied by the dilation factor: (i×dilation, j×dilation); Deformable convolution: Reads (Δx, Δy) from the offset buffer and superimposes it with the base coordinates: ； ； Coordinate quantization: decomposes floating-point coordinates into integer parts (x_int, y_int) and fractional parts (x_frac, y_frac); The interpolation unit circuit includes four integer point reading modules, a weight calculation module, and a multiplication accumulation tree, which is implemented using a four-stage pipeline. The offset cache is a dedicated SRAM, stored in a similar manner to the feature map, and supports parallel reading with the feature map.

8. A tensor kernel-driven online image transcoding circuit for multimodal large models, characterized in that, The circuit is the hardware implementation circuit corresponding to the tensor kernel-driven online image transcoding device for multimodal large models as described in any one of claims 1-7. The circuit is designed in conjunction with the tensor processing unit array and includes a multi-channel GDDR6 interface circuit, an N-way parallel image transcoding hardware circuit, an inter-channel data aggregation hardware circuit, and a direct connection interface circuit for the tensor processing unit.

9. A tensor kernel-driven online image transcoding method for multimodal large models, characterized in that, The method is implemented based on the tensor kernel-driven online image transcoding device for multimodal large models as described in any one of claims 1-7, and includes the following steps: S1. The dynamic configuration parser receives and parses the convolution parameters and storage distribution information, configures the storage volume mapping rules, address generation mode, and rearranges the network routing table. S2. The input feature map is loaded into a multi-bank SRAM array through the direct memory access engine, and the bank data arrangement circuit writes the data into the corresponding bank according to bank_index and bank_addr. S3. The prefetch control circuit calculates the coordinates of the first window, the window data generator generates the global coordinates of the sampling points, and reads data in parallel from the multi-memory SRAM. S4. Determine the convolution type. If it is deformable convolution, perform bilinear interpolation through the interpolation unit and then send the data into the rearrangement network. If it is standard / dilated convolution, send the data directly into the non-blocking rearrangement network. S5. The non-blocking rearrangement network arranges the data in matrix multiplication order and stores it in a ping-pong buffer as the current window data. S6. After the tensor processing unit is ready, push the data to the input register of the tensor processing unit, and the tensor processing unit performs matrix multiplication calculation. S7. While the tensor processing unit performs the matrix multiplication of the current window, the prefetch control circuit prefetches the data of the next window, switches the ping-pong buffer, and repeats steps S3 to S6 until all output positions are processed.

Images