A large model inference method, an intelligent edge device, and a data processing method thereof

Abstract

The present application relates to the technical field of edge device data processing, and discloses a large model inference method, an intelligent edge device and a data processing method thereof, which are designed in coordination with representation optimization and execution optimization in view of the scene of smaller cache and tighter bandwidth on the edge side: the representation structure is naturally adapted to the execution path, which helps to improve effective computing strength and reduce cache jitter in bandwidth-sensitive small batch decoding; secondly, a multi-region shared joint bitmap is used to represent the non-zero activity of multiple regions with a single index structure, and the aligned multi-path compression load flow is used to synchronously access multiple region loads under the same index, thereby reducing index redundancy and indirect addressing levels in structure; finally, the output of parallel decoding is directly organized into the fragment layout required by the matrix multiplication and addition instruction, so that sparse low-bit weights are more directly mapped to the input of the matrix multiplication and addition hardware unit, indirect addressing and repeated decoding / copying are avoided, and higher hardware utilization is supported.

Description

Technical Field

[0001] This invention relates to the field of edge device data processing technology, and in particular to a large model inference method, an intelligent edge device, and a data processing method thereof. Background Technology

[0002] Large model inference (especially Transformer-type large language models) can be applied to various service scenarios when deployed on edge devices. Figure 1 This diagram illustrates the deployment scenarios and service flow of large language models at the edge. It showcases a typical deployment architecture and service call flow for large language models on edge devices (such as smartphones, IoT gateways, and embedded devices). The edge device layer includes locally stored model weights, the inference engine, and hardware accelerators (GPUs, NPUs, DSPs). The service flow can be: user request (e.g., text generation, question answering) → input preprocessing → model inference → post-processing → return result. It can also include an edge-cloud collaborative model, where some large model layers execute at the edge, and complex requests are offloaded to the cloud.

[0003] Large model inference, when deployed on edge devices, is generally constrained by a combination of factors, including GPU / RAM capacity, off-chip bandwidth, power consumption, and latency. Furthermore, the autoregressive decoding stage often exhibits a "small-batch, skinny matrix multiplication" pattern, resulting in low arithmetic strength and performance more readily limited by global memory bandwidth and cache hierarchy rather than pure computational power. Existing research indicates that the key MatMul in generative inference is often a skinny MatMul, whose performance significantly falls below the "memorywall," thus requiring improvements in effective throughput by reducing off-chip reads and writes.

[0004] To reduce model storage and computational overhead, industry and academia commonly employ two main compression techniques: The first is low-bit quantization (e.g., 8-bit, 4-bit, or even 3 / 2-bit weight quantization), with representative works including post-training low-bit quantization methods like GPTQ, and weight quantization strategies aimed at "avoiding hardware-incompatible mixed precision" (such as AWQ and LLM.int8). The common goal of these methods is to reduce the bit width of the weight payload, thereby lowering storage and bandwidth costs. The second technique is pruning / sparseness, especially unstructured pruning of large model weights to construct a large number of zero values; representative works include SparseGPT and Wanda, which have proven to maintain acceptable accuracy at a considerable sparsity rate. These compression techniques gradually transform linear layer computations during inference into sparse matrix computations, making sparse matrix multiplication a key operator.

[0005] However, the combination of low bit depth and unstructured sparseness exposes the structural bottlenecks of traditional sparse representations: traditional CSR / COO formats, which are based on "coordinates / offsets," require explicit storage of row pointers and column indices (and possibly block indices). The bit width of their index arrays does not decrease proportionally with the weighted payload from FP16 / INT8 to INT4 / INT3 / INT2, leading to a sharp increase in the proportion of "metadata" such as indexes / bitmaps / offsets in total memory accesses. A typical CSR representation consists of arrays such as data / indices / indptr, and the index part naturally has fixed-granularity overhead. Practice shows that in low bit depth sparse inference, there is a phenomenon of "smaller payload but higher bandwidth for indexes," which can be characterized by the "payload-metadata ratio (PMR)." This problem is even more pronounced in edge-side small-batch decoding scenarios.

[0006] Meanwhile, to achieve actual acceleration on GPUs, sparse weights need to be "normalized" to hardware-friendly access and computation paths. Existing software stacks provide general-purpose sparse operator libraries (e.g., cuSPARSE supports various sparse storage formats and provides general interfaces such as SpMM / SpMV), but these general formats and kernels often suffer from instability in low-arithmetic-intensity scenarios due to indirect addressing, thread divergence, and cache misses. On the hardware side (represented by NVIDIA), 2:4 structured sparse tensor kernels have been introduced in architectures like Ampere, and high-throughput matrix multiplication and addition support for structured sparses is provided through libraries such as cuSPARSELt. However, this approach imposes strict constraints on the sparse pattern (e.g., 2:4 pattern), and is not equivalent to the sparse distribution obtained through unstructured pruning.

[0007] Existing methods for accelerating large model inference mainly fall into two categories:

[0008] The first category is the "Load-as-Sparse & Compute-as-Dense" approach for generative reasoning: for example, Flash-LLM points out that skinnyMatMul is limited by bandwidth, and proposes to reduce global memory read and write through sparse format, reconstruct it into dense tiles that can be consumed by tensor kernel at runtime, and use software pipeline to overlap data transport and computation; among them, "efficient sparse data format and sparse to dense extraction" are the difficulties in implementation.

[0009] The second category is a framework that co-designs "sparse weighted compressed representation" with "tensor kernel-friendly decoding / pipelining". For example, the Hong Kong University of Science and Technology's public research entry provides a summary description of SpInfer (EuroSys2025), which clearly points out that one of the key reasons why the benefits of unstructured pruning in LLM inference are difficult to realize is "non-zero metadata overhead of the index". It proposes designs such as tensor kernel-oriented bitmap encoding and shared memory bitmap decoding, asynchronous pipelining, etc., and achieves significant speedup compared with existing implementations within a certain sparsity range.

[0010] While the aforementioned technologies have advanced the acceleration of large model inference, unresolved issues remain in scenarios combining low-bit (2 / 3 / 4-bit) data, unstructured sparse data, and edge mini-batch decoding.

[0011] (1) The proportion of index / bitmap metadata is too high: the lower the weight, the smaller the bit and payload, the more "expensive" the metadata appears, the lower the PMR, and the more easily the off-chip bandwidth is swallowed up by the metadata.

[0012] (2) Sparse traversal paths are irregular and cache locality is poor: edge-side caches are smaller and bandwidth is tighter, making them more sensitive to index efficiency and regular memory access.

[0013] (3) Disconnect between online decoding and tensor kernel input fragment assembly: If the compressed representation does not match the operand layout of the hardware instruction, multi-level indirect addressing and repeated decoding / transfer will occur, reducing the utilization of matrix multiplication and addition hardware.

[0014] (4) Relying solely on pipelining optimization is insufficient to fundamentally reduce metadata traffic: Without changing the representation structure, relying solely on kernel scheduling / pipelining is unlikely to reduce index memory accesses at the source. Summary of the Invention

[0015] This invention provides a large model inference method, an intelligent edge device, and a data processing method thereof, which can improve the efficiency of large model inference when deployed on edge devices.

[0016] The first aspect of this invention discloses a large-model inference method, wherein the large model is applied to edge device deployment scenarios, and the method includes:

[0017] Obtain the pruned and quantized weight matrix on the input side;

[0018] The weight matrix is ​​divided into global weight tiles, and the global weight tiles are further divided into local weight tiles, wherein the size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions;

[0019] The weighted local tile is divided into Q regions and a logical joint bitmap is constructed, wherein the positions in the Q regions with non-zero loads have corresponding active bits in the logical joint bitmap;

[0020] Bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity, thereby adjusting the weighted local tile row order; the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions are obtained;

[0021] Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix;

[0022] When the target model performs inference tasks: the target joint bitmap is segmented, and a compressed payload flow access index is generated based on the active bit statistics at the word level, line level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed payload flow using the access index.

[0023] The target weight value is packaged into a register segment that conforms to the matrix multiply-add instruction, and the active local tile is loaded into the register segment. After the multiply-add is completed, the result matrix is ​​output.

[0024] As an optional implementation, in the first aspect of the present invention, the method further includes:

[0025] When the target model performs inference tasks, a shared storage double buffer is set for the weights and activations respectively, and the output accumulation register segment is initialized.

[0026] The online execution phase employs shared storage double buffering and a producer-consumer decoupled pipeline, specifically including: configuring shared storage double buffer read / write areas for the weighted local tiles and the activated local tiles, and setting up at least one producer thread bundle and one or more consumer thread bundles within the parallel thread block. The producer thread bundle asynchronously moves the next weighted local tile or activated local tile to the shared storage write buffer; the consumer thread bundle reads the current tile from the read buffer and performs parallel decoding and computation; the double buffer is rotated to hide memory access latency.

[0027] As an optional implementation, in the first aspect of the present invention, the number of producer thread bundles or consumer thread bundles and the size of the shared memory double buffer read / write area are adjusted according to the thread block size, shared memory capacity and memory access latency.

[0028] As an optional implementation, in the first aspect of the invention, dividing the weighted local tiles into Q regions and constructing a logically joined bitmap includes:

[0029] Set the number of row partitions RA and the number of column partitions CA, and divide the weighted local tile into Q regions, where Q = RA × CA;

[0030] For the Q weight values ​​at the same relative spatial position in the Q regions, construct a logically joined bitmap; wherein, for any relative spatial position (i, j), if there is any non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logically joined bitmap is 1; if there is no non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logically joined bitmap is 0.

[0031] As an optional implementation, in the first aspect of the present invention, the number of row partitions RA = 2; the number of column partitions CA = 2, and the construction of the logical union bitmap includes:

[0032] The weighted local tile is divided into 4 regions. For any relative spatial position (i, j), the corresponding weight value of the 4 regions is taken. If there is any non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logical union bitmap is 1.

[0033] Only the weight values ​​of the four regions corresponding to the position Bt(i,j)=1 are output and compressed for storage, so that the four bitmap information are merged into one logical joint bitmap.

[0034] As an optional implementation, in the first aspect of the invention, the bitmap-aware row rearrangement aimed at minimizing joint bitmap activity, thereby adjusting the weighted local tile row order, includes:

[0035] Using a binary non-zero mask of weighted local tiles as input, define the logical active cost of candidate row groups after column partitioning aggregation, and use dynamic programming of group subsets to recursively solve the row arrangement corresponding to the minimum total cost; output the row arrangement and rearrange the weighted local tiles according to the row arrangement.

[0036] As an optional implementation, in the first aspect of the present invention, the target joint bitmap is segmented, and a compressed payload flow access index is generated based on word-level, row-level, or segment-level active bit statistics. The target weight value is extracted in parallel from the Q-path aligned compressed payload flow using the access index, including:

[0037] The combined bitmap is divided into segments of a preset width G, and threads within a thread bundle are mapped to different segments.

[0038] The base address of the bit segment is obtained by prefix summing at the word level, and the offset within the segment is obtained by counting the active bits within the segment.

[0039] The compressed load stream access index is obtained by combining the bit segment base address and the intra-segment offset;

[0040] The target weight value is obtained by extracting all the Q-path aligned compressed load streams in parallel using the same index.

[0041] As an optional implementation, in the first aspect of the present invention, the edge device is a parallel computing device having a shared storage hierarchy, thread bundle-level scheduling capability, and matrix multiplication and addition hardware unit;

[0042] The data to be inferred includes one or more of the following: voice data, text data, image data, and video data.

[0043] The inference task includes one or more of the following: decoding stage, mini-batch inference stage, bandwidth-sensitive stage, pre-filled inference stage, and hybrid inference stage.

[0044] A second aspect of this invention discloses a data processing method for an intelligent edge device, wherein the intelligent edge device is equipped with a target model, and the method includes:

[0045] Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix;

[0046] When the target model performs inference tasks: the target joint bitmap is segmented, and a compressed payload flow access index is generated based on the active bit statistics at the word level, line level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed payload flow using the access index.

[0047] The target weight value is packaged into a register segment that conforms to the matrix multiply-add instruction, and the active local tile is loaded into the register segment. After the multiply-add is completed, the result matrix is ​​output.

[0048] The intelligent edge device is pre-configured in the following manner:

[0049] Obtain the pruned and quantized weight matrix on the input side;

[0050] The weight matrix is ​​divided into global weight tiles, and the global weight tiles are further divided into local weight tiles, wherein the size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions;

[0051] The weighted local tile is divided into Q regions and a logical joint bitmap is constructed, wherein the positions in the Q regions with non-zero loads have corresponding active bits in the logical joint bitmap;

[0052] Bitmap-aware row rearrangement is performed with the goal of minimizing the active items in the joint bitmap, thereby adjusting the weighted local tile row order; the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions are obtained.

[0053] A third aspect of the present invention discloses an intelligent edge device, wherein the intelligent edge device carries a target model and is pre-configured in the following manner:

[0054] Obtain the pruned and quantized weight matrix on the input side;

[0055] The weight matrix is ​​divided into global weight tiles, and the global weight tiles are further divided into local weight tiles, wherein the size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions;

[0056] The weighted local tile is divided into Q regions and a logical joint bitmap is constructed, wherein the positions in the Q regions with non-zero loads have corresponding active bits in the logical joint bitmap;

[0057] Bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity, thereby adjusting the weighted local tile row order; the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions are obtained;

[0058] And, the target model is used for:

[0059] Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix;

[0060] The target joint bitmap is segmented, and a compressed payload flow access index is generated based on the active bit statistics at the word level, row level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed payload flow using the access index.

[0061] The target weight value is packaged into a register segment that conforms to the matrix multiply-add instruction, and the active local tile is loaded into the register segment. After the multiply-add is completed, the result matrix is ​​output.

[0062] Compared with the prior art, the present invention has the following beneficial effects:

[0063] First, the present invention uses a "multi-region shared logical joint bitmap" to enable a single index structure to represent the non-zero activity of multiple regions, and uses "aligned multiple compressed load streams" to access multiple region loads simultaneously under the same index, thereby structurally reducing index redundancy and indirect addressing levels.

[0064] Second, this invention addresses the scenario of "smaller cache and tighter bandwidth" on the edge side by coordinating the design of "representation optimization (shared joint bitmap, row rearrangement)" and "execution optimization (double buffered pipeline, shared storage parallel decoding, fragment direct generation)": the representation structure is naturally adapted to the execution path, which helps to more robustly improve the effective computation intensity and reduce cache jitter in bandwidth-sensitive small batch decoding.

[0065] Third, the invention's "register fragment direct generation" directly organizes the output of parallel decoding into the fragment layout required by matrix multiplication and addition instructions, making sparse low-bit weights more directly mapped to the input of matrix multiplication and addition hardware units, avoiding multi-layer indirect addressing and repeated decoding / copying in the traditional path, and supporting higher hardware utilization from a mechanism perspective.

[0066] In summary, this invention can improve the efficiency of large model inference when deployed on edge devices. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a flowchart illustrating the deployment scenarios and service flow of large language models on the edge side.

[0069] Figure 2 This is a flowchart illustrating a large model inference method disclosed in an embodiment of the present invention;

[0070] Figure 3(a) is a schematic diagram of the offline stage in a large model-based inference task implementation method disclosed in an embodiment of the present invention;

[0071] Figure 3(b) is a schematic diagram of the online stage in a large model-based inference task implementation method disclosed in an embodiment of the present invention;

[0072] Figure 4 This is a schematic diagram of shared storage double buffering and producer-consumer decoupling pipeline;

[0073] Figures 5(a)-5(c) This is a flowchart illustrating the process of performing parallel decoding of shared memory based on a joint bitmap. Detailed Implementation

[0074] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0075] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0076] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0077] This invention discloses a large model inference method, an intelligent edge device, and a data processing method thereof, which are used to improve the efficiency of large model inference when deployed on edge devices.

[0078] Example 1

[0079] The first embodiment of the present invention aims to solve (at least) the following main technical problems: when performing low-bit sparse large model inference on edge devices, traditional sparse representation leads to excessively high metadata overhead, irregular sparse traversal causes poor cache locality and high decoding overhead, and compressed representation is difficult to efficiently map to matrix multiplication and addition hardware units, resulting in insufficient hardware utilization.

[0080] The main inventive concept of Embodiment 1 of the present invention is as follows:

[0081] Without sacrificing the existing compression benefits of "unstructured sparsity (from pruning)" and "low bit payload (from quantization)," we reduce the bandwidth and latency overhead caused by metadata and indirect addressing by sharing the index structure, offline rearrangement with minimal bitmap activity, and online parallel decoding and pipelined execution for shared memory / register fragments. This allows sparse low bit weights to be consumed more directly by the tensor kernel / matrix engine, thereby improving end-to-end inference performance.

[0082] Specifically, such as Figure 2 As shown, Embodiment 1 of the present invention discloses a large model inference method. This large model is applied to edge device deployment scenarios, wherein the edge device can be a parallel computing device with a shared memory hierarchy, thread bundle-level scheduling capability, and matrix multiplication and addition hardware units; the method may include:

[0083] Step 101: Obtain the pruned and quantized weight matrix on the input side.

[0084] The weight quantization bit width can be one of the low-bit weights such as 2 / 3 / 4 / 8 bits.

[0085] Step 102: Divide the weight matrix into global weight tiles, and then divide the global weight tiles into local weight tiles.

[0086] Specifically, the size of the weighted local tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions. For example, the size of the local tiles is set to match the shared storage capacity of the target hardware, the register size, and the granularity of the matrix multiply-accumulate instruction operands, making the local tiles the basic unit for compression encoding and decoding.

[0087] Step 103: Divide the weighted local tiles into Q regions and construct a logical joint bitmap.

[0088] In this context, the locations with non-zero loads in the Q regions are represented by active bits in the logical union bitmap. Step 103 is used to decompose each local tile into regions. For example, the number of row partitions RA and the number of column partitions CA are set to divide the local tile into Q regions, where Q = RA × CA. This "region" concept is used to organize multiple sub-blocks within the same local tile so that the loads of multiple regions can be accessed in alignment under the same index structure.

[0089] Specifically, for Q low-bit weight values ​​at the same relative spatial location in Q regions, a joint bitmap entry is used to characterize whether "at least one non-zero payload exists in this set of locations". The corresponding entry in the joint bitmap is active as long as the location is non-zero in any region. The key intention of this design is to allow a single bitmap entry to "serve multiple regions," reducing the redundancy of maintaining separate bitmaps / indexes for each region. The low-bit weight values ​​include 2 / 3 / 4 / 8 bits.

[0090] Step 104: Perform bitmap-aware row rearrangement with the goal of minimizing the active terms of the joint bitmap, thereby adjusting the weighted local tile row order; to obtain the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions.

[0091] The objective is to minimize the number of active items in the joint bitmap. Rows within local tiles are rearranged to reduce the number of active items in the joint bitmap without altering the set and values ​​of non-zero elements, thus shortening the subsequent compressed payload stream and reducing bitmap / index overhead. This step is performed offline to avoid increasing online inference complexity. For the Q-way aligned compressed payload stream corresponding to Q regions, the Q-way compressed payload stream is logically aligned so that for any active item in the joint bitmap, the low-bit payload values ​​of multiple regions can be accessed synchronously using the same index, and inactive bits are zero-padded.

[0092] Step 105: Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix.

[0093] The data to be inferred includes one or more of the following: speech data, text data, image data, and video data. Obtaining activation local tiles based on input data (such as speech features and image pixels) is a dynamic, real-time process in online inference. Step 105 may include: converting the original signal into a two-dimensional matrix (or a higher-dimensional tensor) that the model can process, where each row / column of the matrix corresponds to a vector representation of a time step or spatial location. Based on the currently calculated output coordinates, sub-blocks matching the size of the weighted local tiles are dynamically cut from the activation matrix. Subsequently, the cut sub-blocks (possibly quantized) can be placed in a double-buffered area of ​​shared storage, awaiting input into the matrix multiply-add unit along with the decoded weighted local tiles.

[0094] Step 106: When the target model performs inference tasks: the target joint bitmap is segmented, and a compressed load flow access index is generated based on the active bit statistics at the word level, row level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed load flow using the access index.

[0095] In step 106, after extracting the target weight value from the Q-way aligned compressed payload stream in parallel using the access index, it may also include automatically padding inactive bits with zeros. Step 106 organizes for SIMT parallelism and shared memory to reduce divergence and reduce multi-level indirect addressing.

[0096] The inference task may include one or more of the following: decoding stage, mini-batch inference stage, bandwidth-sensitive stage, pre-filled inference stage, and hybrid inference stage.

[0097] Step 107: Pack the target weight values ​​into a register segment that conforms to the matrix multiplication and addition instruction, load the active local tile into the register segment, and output the result matrix after completing the multiplication and addition.

[0098] Thus, efficient representation and inference acceleration of low-bit sparse large model indexes have been achieved. After step 107, multiple output result matrices can be concatenated to obtain a complete output matrix; finally, after passing through all layers of the model (tens to hundreds of layers), the inference result is finally output.

[0099] For example, the reasoning process of an example is: "How is the weather today?" → speech features → the output of the first FFN layer is the intermediate features → through 32 Transformer layers → the last layer outputs logits → softmax → "It's sunny today, the temperature is 25°C".

[0100] Steps 101 to 104 can be completed offline, while steps 105 to 107 need to be completed online.

[0101] In summary, the embodiments of this invention creatively propose: (a) introducing RA×CA region decomposition within local tiles and allowing multiple regions to share a logical joint bitmap; (b) performing bitmap-aware row rearrangement (offline) with the goal of minimizing the active items of the joint bitmap; (c) constructing a "multi-path aligned compressed payload stream" so that a joint bitmap index can simultaneously extract multiple region payloads; and (d) online parallel decoding in shared storage using bit-segment / rank / index methods and directly generating register fragments that match matrix multiplication and addition instructions, thereby achieving a short-path closed loop of "compressed representation → decoding → fragment → hardware".

[0102] The embodiments of the present invention can produce the following unexpected technical effects:

[0103] First, it reduces metadata redundancy and improves payload-metadata efficiency. Traditional "region-independent bitmap / index" requires maintaining bitmap or offset / coordinate information for each region; in low-bit scenarios, the payload bit width decreases while the index bit width remains constant, increasing the proportion of metadata. This invention, through a "multi-region shared logical joint bitmap," allows a single index structure to represent the non-zero activity of multiple regions, and uses "aligned multiplexed compressed payload streams" to synchronously access multiple region payloads under the same index, structurally reducing index redundancy and indirect addressing levels.

[0104] A computable example (not dependent on a specific test platform) is as follows: When Q = RA × CA = 4 (e.g., RA = 2, CA = 2), existing technologies require 4 region bitmaps (or equivalent index structures), while this embodiment compresses them into 1 joint bitmap; without considering the differences in additional offset tables, bitmap-type metadata can theoretically be reduced to nearly 1 / 4 of its original size. Furthermore, bitmap-aware row rearrangement, aiming for "minimum number of active items," reduces the number of bits that are 1 in the joint bitmap, thereby shortening the compressed payload stream length and reducing the number of times rank / index generation is performed on active bits. This combined effect of "shared bitmap + minimization of active items" aims to directly improve PMR-type metrics (payload bit width / metadata bit width) and alleviate bandwidth bottlenecks.

[0105] Second, it improves the effective arithmetic strength and cache locality under bandwidth-sensitive mini-batch inference. This invention addresses edge-side scenarios with "smaller cache and tighter bandwidth" by coordinating "representation optimization (shared joint bitmap, row rearrangement)" with "execution optimization (double-buffered pipelining, shared memory parallel decoding, fragment direct generation)": the representation structure naturally adapts to the execution path, helping to more robustly improve effective computational strength and reduce cache thrashing in bandwidth-sensitive mini-batch decoding.

[0106] Third, it improves the utilization rate of matrix multiplication and addition hardware units and reduces the additional cost of "decoding followed by relocation / rearrangement". The "register segment direct generation" embodiment of this invention directly organizes the output of parallel decoding into the segment layout required by matrix multiplication and addition instructions, so that sparse low bit weights are more directly mapped to the input of matrix multiplication and addition hardware units, avoiding multi-layer indirect addressing and repeated decoding / copying in the traditional path, and supporting higher hardware utilization from a mechanism perspective.

[0107] In an optional embodiment, the method may further include:

[0108] When the target model performs inference tasks, a shared storage double buffer is set for the weights and activations respectively, and the output accumulation register segment is initialized.

[0109] The online execution phase employs a shared memory double buffer and a producer-consumer decoupled pipeline. Specifically, this includes configuring a shared memory double buffer read / write area for weighted local tiles and active local tiles, and setting up at least one producer thread bundle and one or more consumer thread bundles within the parallel thread block. The producer thread bundle asynchronously moves the next weighted local tile or active local tile to the shared memory write buffer; the consumer thread bundle reads the current tile from the read buffer and performs parallel decoding and computation; and the double buffer is rotated to hide memory access latency.

[0110] This optional embodiment employs a decoupled producer-consumer thread bundle and configures a double-buffered read / write area during the execution phase, allowing weight / activation prefetching to overlap with the current tile decoding / computation, thereby reducing the direct impact of memory access latency on decoding throughput; it can overlap data transport and computation, reducing the impact of memory access latency on end-to-end latency.

[0111] In this optional embodiment, the number of producer thread bundles or consumer thread bundles and the size of the shared memory double buffer read / write area are further adjusted according to the thread block size, shared memory capacity and memory access latency.

[0112] In yet another alternative embodiment, the weighted local tiles are divided into Q regions and a logically joined bitmap is constructed, including:

[0113] Set the number of row partitions RA and the number of column partitions CA, and divide the weighted local tile into Q regions, where Q = RA × CA;

[0114] For Q weight values ​​at the same relative spatial position in Q regions, construct a logically joined bitmap; wherein, for any relative spatial position (i, j), if there is any non-zero weight value at position (i, j) in Q regions, then the corresponding item Bt(i, j) in the logically joined bitmap is 1; if there is no non-zero weight value at position (i, j) in Q regions, then the corresponding item Bt(i, j) in the logically joined bitmap is 0.

[0115] In yet another optional embodiment, the number of row partitions RA = 2 and the number of column partitions CA = 2 are set to construct a logical union bitmap, including:

[0116] The weighted local tile is divided into 4 regions. For any relative spatial position (i, j), the corresponding weight values ​​of the 4 regions are taken. If there is any non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logical union bitmap is 1.

[0117] Only the weight values ​​of the four regions corresponding to the position Bt(i,j)=1 are output and compressed for storage, thus merging the four bitmap information into a single logical union bitmap. This optional embodiment merges the four bitmap information into a single union bitmap, thereby reducing the indexing burden.

[0118] In yet another alternative embodiment, bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity, thereby adjusting the weighted local tile row order, including:

[0119] Taking the binary non-zero mask of the weighted local tiles as input, we define the logical active cost of candidate row groups after column partitioning aggregation, and use dynamic programming of group subsets to recursively solve the row permutation corresponding to the minimum total cost; output the row permutation and rearrange the weighted local tiles according to the row permutation.

[0120] In another optional embodiment, the target joint bitmap is segmented, and a compressed payload flow access index is generated based on word-level, row-level, or segment-level active bit statistics. The access index is then used to extract the target weight value from the Q-path aligned compressed payload flow in parallel, including:

[0121] The combined bitmap is divided into segments of a preset width G, and threads within a thread bundle are mapped to different segments.

[0122] The base address of the bit segment is obtained by prefix summing at the word level, and the offset within the segment is obtained by counting the active bits within the segment.

[0123] The compressed load stream access index is obtained by combining the bit segment base address and the intra-segment offset;

[0124] The target weight value is obtained by extracting all Q-path aligned compressed load streams in parallel using the same index.

[0125] Example 2

[0126] Embodiment 2 of this invention proposes a method for implementing inference tasks based on large models. The data to be inferred includes one or more of the following: speech data, text data, image data, and video data. This method is based on a collaborative scheme of "indexed efficient sparse representation + online decoding direct-connected matrix multiplication and addition hardware," which generally includes two major stages: offline representation generation and online inference execution.

[0127] The offline phase (representing generation) is shown in Figure 3(a), and the online phase (inference execution) is shown in Figure 3(b). The two phases are as follows:

[0128] First, obtain the pruned and quantized sparse low-bit weight matrix and activation matrix on the input side; the weight quantization bit width can be one of 2 / 3 / 4 / 8 bits, and the activation can also be represented by the same or different bit widths.

[0129] Second, the weight matrix is ​​divided into hierarchical tiles: the weights are divided into global tiles, and the global tiles are further divided into local tiles. The size of the local tiles is matched and set according to the target hardware's shared memory capacity, register size, and the granularity of matrix multiplication and addition instruction operands, so that the local tiles become the basic units for compression encoding and decoding.

[0130] Third, each local tile is decomposed into regions: The number of row partitions RA and the number of column partitions CA are set, dividing the local tile into Q regions, where Q = RA × CA. This "region" concept is used to organize multiple sub-blocks within the same local tile, enabling aligned access to loads from multiple regions under the same index structure.

[0131] Fourth, construct a shared Union Bitmap / Logical Union Mask: For Q low-bit weight values ​​at the same relative spatial location in Q regions, use a single Union Bitmap entry to characterize whether "at least one non-zero payload exists in this set of locations". The corresponding Union Bitmap entry is active as long as the location is non-zero in any region. The key intention of this design is to allow a single bitmap entry to "serve multiple regions," reducing the redundancy of maintaining separate bitmaps / indexes for each region.

[0132] Fifth, perform Bitmap-aware RowReordering: With the goal of minimizing the number of active items in the joint bitmap, the rows within the local tiles are reordered (can be performed offline). This reduces the number of items set as active in the joint bitmap without changing the set of non-zero elements and their values, thereby shortening the subsequent compressed payload stream and reducing bitmap / index overhead. This step avoids increasing the complexity of online inference by being performed offline.

[0133] Sixth, generate an index-efficient bitmap encoding format, wherein: the encoding includes at least (i) local tile-level offset information (indicating the start position of the payload stream), (ii) logically combined bitmap information, and (iii) multiple compressed payload streams corresponding to Q regions; wherein "the multiple compressed payload streams are logically aligned" so that for any active item in the combined bitmap, the low bit payload values ​​of multiple regions can be accessed synchronously using the same index, and inactive bits are zero-padded.

[0134] Seventh, the online execution phase employs shared storage double buffering and a producer-consumer decoupled pipeline, such as... Figure 4 As shown: a shared storage double-buffered read / write area is configured for the weighted local tile and the active local tile, and at least one producer thread bundle and one or more consumer thread bundles are set in the parallel thread block. The producer asynchronously prefetches the next local tile, and the consumer decodes and calculates the current local tile, thereby overlapping data transportation and calculation and reducing the impact of memory access latency.

[0135] Eighth, perform shared-memory parallel decoding based on the joint bitmap, such as... Figure 5(a) , 5(b) As shown in 5(c): the joint bitmap is divided into bit segments, and a compressed payload stream access index is generated based on the active bits at the word / row / segment level. The active bits are ranked to obtain the segment offset, and the low bit weight values ​​are extracted from the multiple compressed payload streams in parallel using this index. Inactive bits are automatically padded with zeros. This step organizes the parallel and shared memory for SIMT to reduce divergence and reduce multi-level indirect addressing.

[0136] Ninth, the low-bit weights obtained from decoding are packaged into register segments according to the operand layout required by the matrix multiplication and addition hardware unit instructions. At the same time, the active local tile is loaded as the active register segment. Both are input into the matrix multiplication and addition hardware (e.g., tensor kernel) to complete the multiplication and addition, and the result matrix is ​​output / accumulated.

[0137] Example 3

[0138] Embodiment 3 of the present invention discloses yet another method for implementing inference tasks based on large models, including...

[0139] Step S1: Perform pruning and low-bit quantization on the linear layer weights of the target large model to obtain a sparse low-bit weight matrix W; the quantization bit width can be INT4 / INT8 or other low-bit forms, and the activation matrix X can be FP16 / INT8 / INT4, etc.

[0140] Step S2: Divide W into global tiles, and further divide each global tile into local tiles; the size of the local tiles is set according to the target hardware shared memory capacity and the matrix multiplication and addition instruction form, so that they can be cached in shared memory and assembled into matrix multiplication and addition fragments in registers.

[0141] Step S3: Divide each local tile into Q regions according to RA×CA; for any relative position (i, j), if there is any non-zero position in the Q regions, then the joint bitmap Bt(i, j) = 1, otherwise Bt(i, j) = 0.

[0142] Step S4: Perform bitmap-aware row rearrangement offline, adjusting the local tile row order without changing the non-zero set and weight value, so as to minimize or reduce the number of active items in the joint bitmap.

[0143] Step S5: Generate indexed efficient bitmap encoding based on the rearranged local tiles: output joint bitmap, local tile offset array, and Q-path aligned compressed payload stream (store the low bit value of each region at the position Bt=1, and logically pad the position Bt=0 with zeros).

[0144] Step S6: Configure thread blocks according to the output matrix at runtime; set up shared storage double buffers for weights and activations respectively, and initialize the output accumulator register segment.

[0145] Step S7: Employ a producer-consumer thread bundle double-buffered pipeline: the producer thread bundle asynchronously moves the weight / activation of the next local tile to the shared memory write buffer; the consumer thread bundle reads the current tile from the read buffer and decodes and calculates it in parallel; the double buffer is rotated to hide memory access latency.

[0146] Step S8: Shared storage parallel decoding: Segment the joint bitmap; perform active bit statistics and rank numbering to generate an access index; use the same index to synchronously extract the low bit values ​​of Q regions from the multi-payload stream and pad with zeros.

[0147] Step S9: Register Frag Packaging and Matrix Multiply-Add: Assemble the extracted low-bit weights into a fragment FragW that conforms to the operand layout of the matrix multiply-add instruction, assemble the activation tile into FragX, call the matrix multiply-add hardware unit to complete the multiplication and add, and write back to the output.

[0148] In an optional embodiment, the local tile is divided into 4 regions; for any relative position (i, j), the corresponding values ​​of the four regions are taken, and if any is non-zero, then Bt(i, j) = 1; then only the positions where Bt(i, j) = 1 are output and compressed for storage, so that the four bitmap information are merged into a joint bitmap, thereby reducing the indexing burden.

[0149] In another optional embodiment, the logical active cost of candidate row groups after column partitioning aggregation is defined using a local tile binary non-zero mask as input, and the row permutation corresponding to the minimum total cost is recursively solved using dynamic programming of group subsets; the rearranged row permutation is output and the local tiles are rearranged accordingly. This algorithm is executed offline and RA / CA is configurable.

[0150] In another alternative embodiment, the joint bitmap is divided into bit segments of fixed width G, and threads within a thread bundle are mapped to different bit segments; the bit segment base address is first obtained by word-level prefix sum, and then the segment offset is obtained by counting active bits within the segment, and the compressed payload stream access index is obtained by combining them; the same index is used to extract all regional payload streams in parallel, and they are combined in a register into a weighted segment that can be consumed by matrix multiplication and addition hardware.

[0151] In another alternative embodiment, the method for implementing large model-based inference tasks can be deployed on parallel computing devices with shared storage hierarchies, thread bundle-level scheduling capabilities, and matrix multiplication and addition hardware units; it is applicable to the decoding stage, mini-batch inference stage, or bandwidth-sensitive stage of large model inference, and can also be extended to the pre-filled or hybrid inference stage.

[0152] In another optional embodiment, RA and CA are not limited to 2×2 and can be any positive integer partition; the joint bitmap can use bit compression, word-level packing, or equivalent storage; row reordering can use dynamic programming, heuristic, local swapping, graph matching, or learning strategies; the number of producer / consumer thread bundles and the shared memory double buffer configuration can be adjusted according to the thread block size, shared memory capacity, and memory access latency; the low bit width can be extended to 2 / 3 / 4 / 8 bits or other compressed representations; it is suitable for large models such as QKV projection layers, attention output projection layers, and feedforward network linear layers, which are mainly based on sparse low-bit matrix multiplication.

[0153] Example 4

[0154] Embodiment 4 of the present invention discloses a data processing method for an intelligent edge device, wherein the intelligent edge device is equipped with a target model, and the method may include:

[0155] S10. Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix.

[0156] The data to be inferred includes one or more of the following: speech data, text data, image data, and video data. Obtaining activation local tiles based on input data (such as speech features and image pixels) is a dynamic, real-time process in online inference. Step S10 may include: converting the original signal into a two-dimensional matrix (or a higher-dimensional tensor) that the model can process, where each row / column of the matrix corresponds to a vector representation of a time step or spatial location. Based on the currently calculated output coordinates, sub-blocks matching the size of the weighted local tiles are dynamically cut from the activation matrix. Subsequently, the cut sub-blocks (possibly quantized) can be placed in a double-buffered area of ​​shared storage, awaiting input into the matrix multiply-add unit along with the decoded weighted local tiles.

[0157] S20. When the target model performs inference tasks: the target joint bitmap is segmented, and a compressed load stream access index is generated based on the active bit statistics at the word level, row level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed load stream using the access index.

[0158] S30. Pack the target weight values ​​into a register segment that conforms to the matrix multiplication and addition instruction, load the active local tile into the register segment, and output the result matrix after completing the multiplication and addition.

[0159] Thus, efficient representation and inference acceleration of low-bit sparse large model index are achieved. After step S30, multiple output result matrices can be concatenated to obtain a complete output matrix; finally, after passing through all layers of the model (tens to hundreds of layers), the inference result is finally output.

[0160] Among them, the weighted local tiles, the target joint bitmap, and the Q-path aligned compressed load stream can be obtained in advance by the methods in Embodiments 1 to 3 of the present invention. For example, the intelligent edge device can be pre-configured in the following way:

[0161] Obtain the pruned and quantized weight matrix on the input side;

[0162] The weight matrix is ​​divided into global weight tiles, and then further divided into local weight tiles. The size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions.

[0163] The weighted local tile is divided into Q regions and a logical union bitmap is constructed. The corresponding entries in the logical union bitmap for the positions with non-zero loads in the Q regions are active bits.

[0164] Bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity terms, thereby adjusting the weighted local tile row order; the target joint bitmap and Q-path aligned compressed load flows corresponding to Q regions are obtained.

[0165] Embodiment 4 of the present invention discloses a smart edge device, wherein the smart edge device carries a target model and is pre-configured in the following manner:

[0166] Obtain the pruned and quantized weight matrix on the input side;

[0167] The weight matrix is ​​divided into global weight tiles, and then further divided into local weight tiles. The size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions.

[0168] The weighted local tile is divided into Q regions and a logical union bitmap is constructed. The corresponding entries in the logical union bitmap for the positions with non-zero loads in the Q regions are active bits.

[0169] Bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity terms, thereby adjusting the weighted local tile row order; the target joint bitmap and Q-path aligned compressed load flows corresponding to Q regions are obtained.

[0170] The weighted local tiles, target joint bitmap, and Q-path aligned compressed load flow can be preset using the methods described in Embodiments 1 to 3 of this invention.

[0171] Furthermore, the aforementioned target model is used for:

[0172] Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix;

[0173] The target joint bitmap is segmented, and a compressed load stream access index is generated based on the active bit statistics at the word, row, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed load stream using the access index.

[0174] Pack the target weight values ​​into a register segment that conforms to the matrix multiply-add instruction, load the active local tile into the register segment, and output the result matrix after the multiply-add is completed.

[0175] Finally, it should be noted that the above embodiments are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A large-scale model reasoning method, characterized in that, The large model is applied to edge device deployment scenarios, and the method includes: Obtain the pruned and quantized weight matrix on the input side; The weight matrix is ​​divided into global weight tiles, and the global weight tiles are further divided into local weight tiles, wherein the size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions; The weighted local tile is divided into Q regions and a logical joint bitmap is constructed, wherein the positions in the Q regions with non-zero loads have corresponding active bits in the logical joint bitmap; Bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity, thereby adjusting the weighted local tile row order; the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions are obtained; Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix; When the target model performs inference tasks: the target joint bitmap is segmented, and a compressed payload flow access index is generated based on the active bit statistics at the word level, line level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed payload flow using the access index. The target weight value is packaged into a register segment that conforms to the matrix multiply-add instruction, and the active local tile is loaded into the register segment. After the multiply-add is completed, the result matrix is ​​output.

2. The large-model inference method according to claim 1, characterized in that, The method further includes: When the target model performs inference tasks, a shared storage double buffer is set for the weights and activations respectively, and the output accumulation register segment is initialized. The online execution phase employs shared storage double buffering and a producer-consumer decoupled pipeline, specifically including: configuring shared storage double buffer read / write areas for the weighted local tiles and the activated local tiles, and setting up at least one producer thread bundle and one or more consumer thread bundles within the parallel thread block. The producer thread bundle asynchronously moves the next weighted local tile or activated local tile to the shared storage write buffer; the consumer thread bundle reads the current tile from the read buffer and performs parallel decoding and computation; the double buffer is rotated to hide memory access latency.

3. The large-model reasoning method according to claim 2, characterized in that, The number of producer thread bundles or consumer thread bundles and the size of the shared storage double buffer read / write area are adjusted according to the thread block size, shared storage capacity and memory access latency.

4. The large-model inference method according to claim 1, characterized in that, The step of dividing the weighted local tiles into Q regions and constructing a logically joined bitmap includes: Set the number of row partitions RA and the number of column partitions CA, and divide the weighted local tile into Q regions, where Q = RA × CA; For the Q weight values ​​at the same relative spatial position in the Q regions, construct a logically joined bitmap; wherein, for any relative spatial position (i, j), if there is any non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logically joined bitmap is 1; if there is no non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logically joined bitmap is 0.

5. The large-model inference method according to claim 4, characterized in that, Set the number of row partitions RA=2; The number of column partitions CA=2, and the constructed logical union bitmap includes: The weighted local tile is divided into 4 regions. For any relative spatial position (i, j), the corresponding weight value of the 4 regions is taken. If there is any non-zero weight value at position (i, j) in the Q regions, then the corresponding item Bt(i, j) in the logical union bitmap is 1. Only the weight values ​​of the four regions corresponding to the position Bt(i,j)=1 are output and compressed for storage, so that the four bitmap information are merged into one logical joint bitmap.

6. The large-model inference method according to claim 1, characterized in that, The bitmap-aware row rearrangement aimed at minimizing joint bitmap activity, thereby adjusting the weighted local tile row order, includes: Using a binary non-zero mask of weighted local tiles as input, define the logical active cost of candidate row groups after column partitioning aggregation, and use dynamic programming of group subsets to recursively solve the row arrangement corresponding to the minimum total cost; output the row arrangement and rearrange the weighted local tiles according to the row arrangement.

7. The large-model inference method according to claim 1, characterized in that, The target joint bitmap is segmented, and a compressed payload flow access index is generated based on word-level, row-level, or segment-level active bit statistics. The target weight value is then extracted in parallel from the Q-path aligned compressed payload flow using the access index, including: The combined bitmap is divided into bit segments according to a preset width G, and threads within a thread bundle are mapped to different bit segments. The base address of the bit segment is obtained by prefix summing at the word level, and the offset within the segment is obtained by counting the active bits within the segment. The compressed load stream access index is obtained by combining the bit segment base address and the intra-segment offset; The target weight value is obtained by extracting all the Q-path aligned compressed load streams in parallel using the same index.

8. The large-model reasoning method according to claim 1, characterized in that, The edge device is a parallel computing device with a shared storage hierarchy, thread bundle-level scheduling capability, and matrix multiplication and addition hardware unit; The data to be inferred includes one or more of the following: voice data, text data, image data, and video data. The inference task includes one or more of the following: decoding stage, mini-batch inference stage, bandwidth-sensitive stage, pre-filled inference stage, and hybrid inference stage.

9. A data processing method for an intelligent edge device, characterized in that, The intelligent edge device is equipped with a target model, and the method includes: Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix; When the target model performs inference tasks: the target joint bitmap is segmented, and a compressed payload flow access index is generated based on the active bit statistics at the word level, line level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed payload flow using the access index. The target weight value is packaged into a register segment that conforms to the matrix multiply-add instruction, and the active local tile is loaded into the register segment. After the multiply-add is completed, the result matrix is ​​output. The intelligent edge device is pre-configured in the following manner: Obtain the pruned and quantized weight matrix on the input side; The weight matrix is ​​divided into global weight tiles, and the global weight tiles are further divided into local weight tiles, wherein the size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions; The weighted local tile is divided into Q regions and a logical joint bitmap is constructed, wherein the positions in the Q regions with non-zero loads have corresponding active bits in the logical joint bitmap; Bitmap-aware row rearrangement is performed with the goal of minimizing the active items in the joint bitmap, thereby adjusting the weighted local tile row order; the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions are obtained.

10. A smart edge device, characterized in that, The intelligent edge device carries the target model and is pre-configured in the following manner: Obtain the pruned and quantized weight matrix on the input side; The weight matrix is ​​divided into global weight tiles, and the global weight tiles are further divided into local weight tiles, wherein the size of the local weight tiles is matched with the shared storage capacity of the edge device and the matrix multiply-accumulate instructions; The weighted local tile is divided into Q regions and a logical joint bitmap is constructed, wherein the positions in the Q regions with non-zero loads have corresponding active bits in the logical joint bitmap; Bitmap-aware row rearrangement is performed with the goal of minimizing joint bitmap activity, thereby adjusting the weighted local tile row order; the target joint bitmap and the Q-path aligned compressed load flow corresponding to the Q regions are obtained; And, the target model is used for: Obtain the data to be inferred, obtain the activation matrix based on the data to be inferred, and dynamically cut out the activation local tiles that match the size of the weighted local tiles from the activation matrix; The target joint bitmap is segmented, and a compressed payload flow access index is generated based on the active bit statistics at the word level, row level, or segment level. The target weight value is extracted in parallel from the Q-way aligned compressed payload flow using the access index. The target weight value is packaged into a register segment that conforms to the matrix multiply-add instruction, and the active local tile is loaded into the register segment. After the multiply-add is completed, the result matrix is ​​output.

Images