A hierarchical factorization-based end-side large model kv cache reuse and compression method

Abstract

The present disclosure is a hierarchical factorization end-side large model KV cache reuse and compression method, comprising: data acquisition division; factorization cache unit construction; offline model calibration; online factor rank compression; local perception layer-by-layer factor reuse. The embodiment is based on the key-value (KV) factorization cache expression, splits the keys and values of the visual tokens in each network layer of the visual language model, and describes them jointly by the head mixed factor and the feature factor; performs online factor rank compression and local perception layer-by-layer factor reuse in the factor space respectively, and reconstructs into a standard KV tensor before entering attention calculation. The internal cache management mode of the present application is transparent to the inference backend, does not require modification of the model decoding process, attention mask semantics or user-visible context order. Reduces KV cache occupation, reduces continuous visual input construction overhead, and avoids destroying attention routing.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a hierarchical factorization method for reusing and compressing end-side large model KV cache. Background Technology

[0002] In recent years, multimodal large-scale models have gradually moved from cloud-based question answering and offline analysis to edge devices such as mobile phones, smart glasses, in-vehicle terminals, and edge robots. In these scenarios, visual language assistants not only need to understand the current image or video clip, but also need to retain visual history over a long period of time during continuous observation, so as to recall the scene, locate evidence, and generate natural language answers when the user asks a delayed question. Transformer-structured visual language models typically write visual lexics into the key-value caches of each network layer through a pre-filling stage, and then reuse these KV caches in the decoding stage to avoid repeatedly calculating historical context.

[0003] However, the video memory, unified memory, memory bandwidth, and power budget of edge devices are all significantly limited. As the visual stream length, the number of spatial blocks, and the number of network layers increase, the historical visual key-value (KV) cache accumulates linearly or nearly linearly, quickly becoming the main resource bottleneck in long-memory inference. Existing visual lexical pruning, merging, sparse attention, or fixed-budget caching methods typically use complete lexical units or complete KV entries as the basic object, which can reduce the number of entries but easily loses fine-grained historical evidence; existing low-rank compression or quantization methods mostly use entire layers or tensors as the object, making it difficult to adapt to the different sensitivities of different layers, different key-value sides, and different rank components to the model output.

[0004] On the other hand, there is often significant redundancy in adjacent frames, adjacent spatial blocks, and similar object regions in continuous visual input. Directly re-caching the complete key-value pair for each visual word would result in redundant calculations and writes. If the complete key-value pair is simply reused, the key-side routing information specific to the current position may be copied along with the shareable visual content, causing a shift in the attention positional relationship.

[0005] Furthermore, edge deployment requires cache management algorithms with low control overhead and good backend compatibility. If the visual encoder needs to be rerun or the entire KV cache needs to be moved every time memory exceeds the limit, the saved GPU memory may be offset by additional latency and energy consumption; if the compressed cache format cannot be directly used by existing efficient attention operators, it will increase the cost of modifying the edge inference framework.

[0006] Therefore, it is necessary to improve one or more of the problems existing in the above-mentioned related technical solutions.

[0007] It should be noted that this section is intended to provide background or context for the technical solutions of this disclosure as set forth in the claims. The description herein does not constitute an admission that it is prior art simply because it is included in this section. Summary of the Invention

[0008] The purpose of this invention is to provide a hierarchical factorization-based end-side large model KV cache reuse and compression method, thereby overcoming at least to some extent one or more problems caused by the limitations and defects of related technologies.

[0009] This invention first provides a hierarchical factorization-based method for reusing and compressing end-side large model key-value caches, including: S1, Data Acquisition and Division: Acquire visual data from the edge and divide the visual data into multiple visual blocks, and further divide each visual block into multiple spatial blocks; S2, Factorization Cache Unit Construction: Input all spatial blocks into the visual language model, use the visual language model to convert the spatial blocks into visual lexical units, and then perform factorization cache on the key and value of the visual lexical units so that the key and value of the visual lexical units are represented by factorization cache units. The factorization cache unit includes four types of visual factors: key-side head mixing factor, key-side feature factor, value-side head mixing factor, and value-side feature factor. S3, Offline Model Calibration: Statistically sort the contribution energy of the rank components corresponding to different network layers and key-value sides of the visual language model, and determine the upper bound of the resident rank, the lower bound of the resident rank, and the sensitivity weight of each layer's cache. S4, Online Factor Rank Compression: Real-time monitoring of the total physical residency of the visual factor cache. When the total physical residency exceeds the trigger threshold, a rank reduction operation is performed on the residency rank of the layer-side cache. S5, Local Perception Layer-by-Layer Factor Reuse: A representative word pool is preset. During the continuous input of visual words, fresh words from the visual words are stored in the representative word pool. Based on the time step and spatial block coordinates of the current visual word, candidate words are selected from the representative word pool to form a local candidate set. The cosine similarity between the current visual word and the visual embedding of each candidate word is calculated to obtain the representative word. The representative word is calibrated offline to obtain the layer-by-layer reuse mask. The layer-by-layer reuse mask is used for factor slot filling.

[0010] In this invention, in S2, the key-side feature factor is injected with rotation position encoding before being written to the cache to obtain a key-side feature factor carrying the current position routing information; the key of a visual word is represented by a key-side head hybrid factor and a key-side feature factor, the value of a visual word is represented by a value-side head hybrid factor and a value-side feature factor, and the factorized cache unit is represented by a key-side head hybrid factor, a key-side feature factor processed by rotation position encoding, a value-side head hybrid factor, and a value-side feature factor.

[0011] In this invention, the visual language model is equipped with a cache manager, and the factorization cache unit is stored in the cache manager. The cache manager is used to record the source tag of each visual word and read or reconstruct the corresponding visual factor according to the source tag. The source tag is used to distinguish whether the visual factor representing the current visual word is an own factor, a factor obtained by reusing a representative word, or a factor that no longer resides physically due to rank reduction.

[0012] In this invention, step S4 involves performing a rank reduction operation on the resident rank of the layer-side cache, including: The amount of memory to be released is calculated based on the number of visual lexical units, the current inhabited rank, the target inhabited rank, and the page-level storage cost corresponding to a visual lexical unit on a rank component. The representation loss is obtained based on the energy of the discarded rank components, the number of visual lexical units, and the layer-side sensitivity weights. The priority measure of candidate rank reduction operations is obtained based on the expected amount of memory released from the discarded rank components and the representation loss. Candidate de-ranking operations are sorted from smallest to largest according to priority metric, and de-ranking operations with lower priority metrics are executed first, until the cumulative release amount reaches the amount of memory to be released or all layer-side caches have reached the corresponding lower bound of the resident rank.

[0013] In this invention, the target resident rank is determined by: setting the intermediate value between the current resident rank and the lower bound of the resident rank as the target resident rank, or by adaptively determining the value based on the amount of memory to be released.

[0014] In this invention, step S4 involves performing a rank reduction operation on the resident rank of the layer-side cache, including: Visual factor caches are addressed according to network layer, key value side, rank component index and lexical page. A group of visual factors with the same layer side and the same rank component are organized into page-level management units. For the key-side page, the page includes the key-side header mixing factor corresponding to the rank component and the key-side feature factor after rotation position encoding; For the value-side page, the page includes the value-side header mixing factor and the value-side feature factor corresponding to the rank component; When the residency rank of the target layer cache is reduced to the target residency rank, pages with rank indexes greater than the target residency rank are released directly, and the residency rank metadata of that layer is updated.

[0015] In this invention, in step S5, if the local candidate set is empty, or the highest cosine similarity is lower than the preset reuse threshold, the current visual word element is represented by the reconstructed visual factors, and after the current visual block processing is completed, the current visual word element is placed into the representative word element pool as a fresh free word element.

[0016] In this invention, in step S5, the layer-by-layer multiplexing mask includes a first layer set and a second layer set. The first layer set is used to control whether the key-side head mixing factor and the value-side head mixing factor can be reused; the second layer set is used to control whether the value-side feature factor can be reused.

[0017] In this invention, the collaborative process of online factor rank compression and local sensing layer-by-layer factor reuse includes: For visual lexical units newly entering the visual language model, their factor slot sources are determined based on the local candidate set and the layer-by-layer reuse mask, and they are written into the factor cache with the current side-resident rank of each layer as a constraint. For historical visual terms that have already been resident, release the pages of low-contribution-rank components based on memory pressure and the priority of candidate de-ranking operations.

[0018] The present invention further provides a hierarchical factorized end-side large model KV cache reuse and compression system, comprising: The visual input module is used to divide visual data into multiple visual blocks; The visual embedding module is used to generate visual embeddings for generating candidate selections based on visual chunking. The factorization cache construction module is used to construct four types of visual factors: key-side head blending factor, key-side feature factor, value-side head blending factor, and value-side feature factor. The lexical management module is used to maintain a fresh, proprietary lexical pool and perform local candidate searches. An online rank compression module is used to select the rank reduction operation based on the trigger threshold, target budget, and memory loss ratio; The pagination cache management module is used to perform page release and resident rank metadata updates; The KV reconstruction module is used to restore visual factors to standard KV tensors before attention calculation.

[0019] The technical solution provided by this invention may include the following beneficial effects: This invention presents a hierarchical factorized end-side large-scale model KV cache reuse and compression method. Based on key-value KV factorized cache representation, it decomposes the keys and values ​​of visual lexical units in each network layer of the visual language model, which are jointly described by head mixing factors and feature factors. Within this factor space, online factor rank compression and local perceptual layer-by-layer factor reuse are performed, and the result is reconstructed into a standard KV tensor before entering attention computation. The internal cache management method of this invention is transparent to the inference backend and does not require modification of the model decoding process, attention mask semantics, or the user-visible context order. This reduces KV cache usage, decreases the overhead of constructing continuous visual inputs, and avoids disrupting attention routing. Attached Figure Description

[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0021] Figure 1 A flowchart illustrating the end-side large model KV cache reuse and compression method of hierarchical factorization in an exemplary embodiment of this disclosure is shown. Figure 2 A flowchart illustrating a hierarchical factorization-based end-side large model KV cache reuse and compression method in yet another exemplary embodiment of this disclosure is shown. Figure 3 This diagram illustrates loss-aware rank reduction in online factor-rank compression in an exemplary embodiment of this disclosure. Figure 4 This diagram illustrates the selection of representative terms in the local spatiotemporal neighborhood during local perceptual layer-by-layer factor reuse in an exemplary embodiment of this disclosure. Detailed Implementation

[0022] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0023] Furthermore, the accompanying drawings are merely illustrative diagrams of embodiments of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities.

[0024] This example implementation first provides a hierarchical factorization-based end-side large model KV cache reuse and compression method. Please refer to [reference needed]. Figure 1 and Figure 2 This method may include: S1-S5, as follows: S1, Data Acquisition and Division: Acquire visual data from the edge and divide the visual data into multiple visual blocks, and further divide each visual block into multiple spatial blocks; S2, Factorization Cache Unit Construction: Input all spatial blocks into the visual language model, use the visual language model to convert the spatial blocks into visual lexical units, and then perform factorization cache on the key and value of the visual lexical units so that the key and value of the visual lexical units are represented by factorization cache units. The factorization cache unit includes four types of visual factors: key-side head mixing factor, key-side feature factor, value-side head mixing factor, and value-side feature factor. S3, Offline Model Calibration: Statistically sort the contribution energy of the rank components corresponding to different network layers and key-value sides of the visual language model, and determine the upper bound of the resident rank, the lower bound of the resident rank, and the sensitivity weight of each layer's cache. S4, Online Factor Rank Compression: Real-time monitoring of the total physical residency of the visual factor cache. When the total physical residency exceeds the trigger threshold, a rank reduction operation is performed on the residency rank of the layer-side cache. S5, Local Perception Layer-by-Layer Factor Reuse: A representative word pool is preset. During the continuous input of visual words, fresh words from the visual words are stored in the representative word pool. Based on the time step and spatial block coordinates of the current visual word, candidate words are selected from the representative word pool to form a local candidate set. The cosine similarity between the current visual word and the visual embedding of each candidate word is calculated to obtain the representative word. The representative word is calibrated offline to obtain the layer-by-layer reuse mask. The layer-by-layer reuse mask is used for factor slot filling.

[0025] In this embodiment, based on the key-value (KV) factorized cache representation, the keys and values ​​of visual lexical units in each network layer of the visual language model are decomposed and described by head mixing factors and feature factors. Online factor rank compression and local perceptual layer-by-layer factor reuse are performed within this factor space, and the resulting data is reconstructed into a standard KV tensor before attention computation. The internal cache management method of this invention is transparent to the inference backend and does not require modification of the model decoding process, attention mask semantics, or the user-visible context order. This reduces KV cache usage, decreases the overhead of constructing continuous visual inputs, and avoids disrupting attention routing.

[0026] The specific process of each step in the above embodiments will be described below.

[0027] S1 collects visual data from the edge device, divides the visual data into multiple visual blocks arranged in chronological order, and further divides each visual block into multiple spatial blocks. The edge device includes a camera or screen capture unit, a processor, memory, and a visual language model running on the processor. The visual language model can be a Transformer architecture multimodal model with various input images / videos and output text, such as Qwen3-VL-2B, GLM-Edge-V-2B, and InternVL2-2B.

[0028] S2, the visual language model (which includes multiple Transformer network layers) encodes visual lexics during the pre-filling stage. For the first lexicographical unit in the input visual sequence... t The visual word element and the first l Each network layer no longer treats its cached objects as indivisible complete key and value tensors. Instead, it adopts a tensor product attention approach, representing keys and values ​​as the sum of tensor products of head mixing factors and feature factors of multiple rank components. The key-side head mixing factor is denoted as... Key-side feature factors are denoted as The side-head mixing factor is denoted as The characteristic factor on the value side is denoted as The key-side feature factor is injected with rotational position encoding before being written to the cache to obtain a key-side feature factor carrying the current position routing information. Its basic form is as follows:

[0029]

[0030]

[0031] in, t This represents the word index in the visual input sequence. l Indicates the Transformer network layer index. i Indicates the rank component index; This indicates that the t-th visual word is in the t-th position. l The bond tensor formed by the layer Indicates the first l The total number of rank components used in the layer bond side factorization representation. Indicates the first t The visual word element in the first l Layer i Head mixing factor on each key lateral rank component This represents the corresponding key-side feature factor; Indicates the first t The visual word element in the first l The value tensor formed by the layer, Indicates the first l The stratified side factorization represents the total number of rank components used. Indicates the first t The visual word element in the first l Layer i Head mixing factor on each lateral rank component. This represents the corresponding value-side characteristic factor; symbol Σ represents the tensor product between the head mixing factor and the eigenfactor, and Σ represents the summation over all rank components. and Represents the normalization coefficient; Indicates the first t The visual word element in the first l Layered factorized cache units, This represents a set or matrix consisting of the mixing factors of each bond side. This represents a set or matrix composed of the characteristic factors of each bond side. This represents the key-side feature factors that have been injected with rotation position encoding; This represents a set or matrix composed of the mixing factors of each value. It represents a set or matrix composed of characteristic factors of each value side.

[0032] In the above expression, and It can also represent the rank hyperparameters on the key side and the value side, respectively. A Class factors are used to mix attention heads or head groups. B Class factors are used to describe the content of each rank component in the feature dimension.

[0033] Factorized cache unit Depend on After rotation position encoding processing , and The process involves reconstructing the standard key and value tensors based on the currently retained, reused, or down-ranked visual factors before the model computes attention, ensuring that the backend receives tensor shapes consistent with the original model. This factorized caching provides fine-grained memory management objects while avoiding exposing compression and reuse strategies to the backend inference implementation.

[0034] In its implementation, factorized cache units can be stored in the edge cache manager as alternative representations of logical key-value (KV) entries. The cache manager records for each visual lexical its visual block, time step, spatial block coordinates, lexical index, resident rank at each level, and source marker for each factor slot. This source marker distinguishes between factors inherent to the current lexical, factors obtained from reused representative lexicals, and factors no longer physically resident due to rank reduction. When the backend requests a batch of KV cached visual lexical units, the cache manager reads or reconstructs the corresponding factors based on the source marker; for parts that have already been downgraded, reconstruction only occurs within the current resident rank. This metadata structure allows compression, reuse, and standard KV reconstruction to be performed in a coordinated manner within the same cache space.

[0035] In addition, the memory of the edge device can maintain three types of data structures. The first type is a logical lexicon table, which records the lexicon number, time step, spatial block coordinates, visual block to which it belongs, and whether it is a fresh, self-owned lexicon. The second type is a factor page table, which records the physical address, reference status, and whether it is still in the resident rank range of the pages corresponding to each layer, each key-value side, and each rank component. The third type is a representative lexicon index, which is used to quickly filter candidate representative lexicons according to the time window and spatial neighborhood. The above data structures can be stored in a unified memory, or they can be stored in the CPU memory and accelerator video memory respectively, depending on the capabilities of the edge platform.

[0036] S3, during the offline preparation phase, the server or training device first performs factorization adaptation on the visual language model, enabling it to generate factorized KV cache while maintaining standard attention output. Subsequently, the system statistically analyzes the contributions of different network layers, different key-value pairs, and different rank components to the output on calibration samples, obtaining the rank component energy order, upper bound of the resident rank, lower bound of the resident rank, and layer-side sensitivity weights. Simultaneously, the system statistically analyzes the visual factor similarity of similar visual word pairs across different network layers, determining the visual embedding reuse threshold, local temporal window, spatial neighborhood range, and first-layer set. Second layer set Metadata such as this is written to the endpoint device as lightweight control parameters during the online execution phase.

[0037] In offline model calibration, the upper bound of the resident rank is used to describe the state of the complete or near-complete cache, the lower bound of the resident rank is used to ensure that the compressed cache still retains the necessary representational capabilities, and the sensitivity weight is used to reflect the degree of impact on the model output after the cache at this layer is downranked.

[0038] S4, based on the aforementioned factorized caching space, the first core technical solution proposed in this application is online factor rank compression. This online factor rank compression targets factor units (i.e., factorized caching units) already written into the visual history cache, aiming to gradually reduce the physical resident rank (removing lower ranks) according to the importance of layer, key-value side, and rank component without directly deleting historical visual lexical units. For example... Figure 3 As shown, value-side factor discarding components and key-side factor discarding components are generated in layer 1, and value-side factor discarding components and key-side factor discarding components are generated in layer 2. All of these components are then deleted.

[0039] During the online inference phase, the edge device continuously maintains the physical residency of the visual factor cache and maintains the current residency rank for each layer-side cache. When the total residency does not exceed the trigger threshold, the system does not perform rank reduction to avoid additional overhead caused by frequent adjustments; when the total residency exceeds the trigger threshold, the system uses the difference between the target budget and the current residency as the amount of memory to be released, and generates all candidate rank reduction operations whose current residency rank is higher than the lower bound of the residency rank.

[0040] The trigger threshold and target budget can be dynamically adjusted based on the real-time resource status of the edge device. For example, when the device is under high load or has limited power, the system lowers the target budget to release more historical visual cache; when the device is connected to power or has high available memory, the system increases the target budget to retain more rank components. The temporal window and spatial neighborhood of the local candidate set (i.e., Figure 4 The local spatiotemporal neighborhood (in the image) can also be adjusted according to the speed of visual flow changes: expanding the candidate range to improve reuse rate when scene changes are slow, and narrowing the candidate range to reduce the risk of misuse during fast movement or scene switching. The above parameter changes do not affect the definition of the factorized cache unit, nor do they change the backend standard KV interface.

[0041] For each candidate de-ranking operation, the system estimates the number of pages that can be released after reducing the corresponding layer's rank from the current rank to the target rank, and estimates the representation loss based on the energy of the discarded rank component, the number of visual lexical units, and the layer's sensitivity weight.

[0042] The expected amount of memory to be freed for the discarded rank component and the priority of the candidate rank reduction operation are represented as follows:

[0043]

[0044] in, l Indicates network layer index, s This represents the layer-side cache index, which is used to distinguish the first... l Key-side cache or value-side cache in the layer; Indicates the first l Layer s The expected amount of physical memory released after the side cache performs a candidate de-ranking operation is the expected amount of memory released after the discarded rank component is executed, and is referred to as the expected release amount. This represents the set of visual lexical units currently residing in the cache. This indicates the number of visual lexical units in the set; Indicates the first candidate operation before execution. l The current resident rank of the cache on the s-th side of layer , represents the target inhabit rank corresponding to the candidate operation; c represents the page-level storage cost of a visual lexical on a rank component, or the unit component storage cost after page discounting in the implementation; Indicates the first l Layer s The representation loss is estimated when the side cache is reduced from the current resident rank to the target resident rank. This representation loss is determined by the contribution energy of the discarded rank component and the corresponding layer side sensitivity. This represents the priority metric for candidate de-ranking operations. Its value is the ratio of representation loss to expected release amount. The smaller the value, the lower the representation loss per unit of memory released.

[0045] The system places all candidate rank reduction operations into a minimum priority queue and executes them first. The rank reduction operation is performed at a lower level until the cumulative release amount reaches the amount of memory to be released, or all layer-side caches have reached the lower bound of their corresponding resident rank. Through this loss-aware greedy selection method, this invention avoids uniformly pruning all layer-side caches and also avoids performing indiscriminate compression based solely on memory size.

[0046] The target resident rank can be set to the intermediate value between the current resident rank and the lower bound of the resident rank, or it can be adaptively determined based on the amount of memory to be released. Regardless of the method used, the target resident rank after the candidate rank reduction operation must not be lower than the lower bound of the resident rank calibrated offline. For historical components that have already been released, this invention does not require regeneration on the end side, thereby ensuring the one-way release characteristic and low online overhead of the compression operation.

[0047] After the candidate de-ranking operation is executed, the system synchronously updates the resident rank information in the logical lexicon table and the factor page table. Released pages are marked as free or reclaimable, and pages still within the resident rank range retain their original physical addresses. Subsequent new factor pages can prioritize reusing reclaimed pages.

[0048] To enable efficient online rank reduction on the client side, this invention further sets up a pagination view based on the rank component's main order to locate the page to be released. Traditional cache layouts are typically organized by lexicon as the main order, meaning a visual lexicon is stored adjacently in the KV states of each layer. This necessitates extensive relocation or rearrangement when releasing pages by rank component. This invention addresses the visual factor cache according to network layer, key-value side, rank component index, and lexicon page. A group of visual factors with the same layer and the same rank component are organized into page-level management units. For key-value side pages, the page includes the corresponding rank component... and position-encoded For the value-side page, the page includes the corresponding rank component. and When the residency rank of a certain layer's cache is reduced to the target residency rank, the system directly releases pages with rank indices greater than the target residency rank and updates the residency rank metadata of that layer (updating the layer's residency rank metadata to the target residency rank). Subsequently written new visual terms are directly reconstructed according to the updated residency rank, while existing terms participate in KV tensor recovery according to their current residency rank during the reconstruction phase. This process does not require reconstructing remaining factors and does not change the position of visual terms in the logical sequence.

[0049] The local perceptual layer-by-layer factor reuse of S5 is the second core technical solution of this invention. Local perceptual layer-by-layer factor reuse is oriented towards new visual blocks in continuous visual input. Its goal is to skip redundant factor calculations and reduce repeated factor reconstruction and repeated cache writing when visual redundancy exists in adjacent time steps or adjacent spatial locations.

[0050] The system maintains a representative lexical pool for pre-filled visual blocks. This pool only accepts fresh, self-owned lexical units. A fresh, self-owned lexical unit refers to one whose reusable visual factors are calculated entirely by itself through a standard factor construction path, rather than being reused from other visual lexical units. This restriction prevents chained reuse from causing errors to propagate along multiple lexical units and ensures that each candidate lexical unit in the representative lexical pool has a traceable, authentic factor source. It should be noted that the representative lexical units in this application are also referred to as representative terms.

[0051] During online operation, before each new visual chunk begins, the system freezes the current representative lexical pool, ensuring that lexical units in this chunk can only find representative lexical units from historical chunks that have already been pre-filled. For the current visual lexical unit... t The system selects candidate words from the representative word pool that are spatially adjacent within the most recent time range based on their time step and spatial block coordinates, forming a local candidate set. Subsequently, the system calculates the input visual embedding of the current word. Visual embeddings for each candidate lexical element The cosine similarity between them is used to select the candidate word with the highest similarity as the representative word.

[0052] To reduce the overhead of representative term search, the representative term index can be organized into a bucket structure based on time steps and spatial block coordinates. When processing the current visual term, the system only accesses candidates in the nearest time step and neighboring spatial buckets; when the number of candidates exceeds the upper limit, candidate terms that are closer in both time and space are prioritized.

[0053] If the local candidate set is empty, or the highest similarity is lower than the preset reuse threshold, the current word element reconstructs all visual factors according to the standard path and enters the representative word element pool as a fresh word element after the current visual block processing is completed. If there is a representative word element that meets the preset reuse threshold, the system enters the layer-by-layer factor slot filling process. The representative word element selection can be expressed as:

[0054] In the above formula for selecting representative lexical units, t This represents the visual word group currently being processed. Index representing candidate representative terms; Indicates based on the current word element tThe local candidate set obtained by filtering by time step and spatial block location Indicates candidate word elements It belongs to this local candidate set; Indicates the current word element t Input visual embedding, Indicates candidate word elements Input visual embedding; express and Cosine similarity between them; argmax represents the similarity in the candidate set. Select candidate words that maximize cosine similarity; This indicates the best representative word element selected at the end. If Empty, or If the highest cosine similarity is lower than the preset reuse threshold, the current visual word element will not be reused as a representative word element.

[0055] After a representative term is accepted, the system does not reuse its complete key-value (KV) state. Instead, it performs factor slot filling based on the layer-by-layer reuse mask obtained offline. The layer-by-layer reuse mask includes the first layer set. Second layer set The first layer set is used to control the key-side mixing factor. Sum of side mixing factors Whether it can be reused, the second-level set is used to control the value-side feature factors. Whether it can be reused. The corresponding layer-by-layer reuse mask is defined as follows:

[0056] in, and All are binary mask variables; This indicates an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise. Indicates that reuse is allowed. and The set of network layers with factor slots Indicates the first l The layer belongs to this set, at this time The value is 1; Indicates that reuse is allowed. The set of network layers with factor slots Indicates the first l The layer belongs to this set, at this time The value is 1. When the value is 1, the current visual word is in the th order. l Layer and It can be filled by the corresponding factors of the representative word element; when When the value is 1, the current visual word is in the th order. l Layer It can be filled by the corresponding factors of the representative word element. It is not controlled by the reuse mask, but is always reconstructed by the current visual lexical itself and injected with the current position encoding.

[0057] When the current layer l belong At that time, the current visual word unit and Factor slots are filled with factors corresponding to representative lexical units; when the current layer l Not belonging to hour, and Reconstructed from the current word element itself. When the current layer... l belong At that time, the current visual word unit Factor slots are filled with factors corresponding to representative lexical units; when the current layer l Not belonging to hour, Reconstructed from the current visual lexical unit itself. Regardless of whether the current layer belongs to any reuse set. All values ​​are always calculated by the current visual word itself, and the rotation position encoding is injected before being written to the cache. This is because... Directly participating in key-side position routing, if the representative word... Copying the entire visual word to the current word can cause attention to incorrectly map the current visual word to the position representing the word. Therefore, this invention separates the shareable local visual content from the position-sensitive key-side routing information.

[0058] Layer-by-layer reuse masks can be automatically generated through offline statistics. For word pairs with high visual similarity, the system compares each layer separately. , and The similarity of factors is used, combined with changes in model output, to determine which network layers and which factor slots are suitable for reuse. Typically, factor slots carrying more local appearance or stable semantics can be included in the reuse set, while factor slots more sensitive to position, occlusion, fine-grained actions, or question-related semantics remain as they are for the current lexical. Because the mask is set layer-by-layer rather than uniformly across the entire model, this invention can adapt to representational differences across different layers; and because the mask is set by factor slot rather than by complete key-value entries, this invention can reduce construction overhead while preserving necessary lexical specificity.

[0059] When a user submits a question and enters the text decoding stage, the standard key tensor and value tensor are reconstructed based on the current visual factor state for each visual lexical, each layer, and each key-value side. For fully reconstructed visual factors, the system directly reads their cached content; for factor slots filled with representative lexicals, the system reads the visual factor corresponding to the representative lexical; for historical caches that have been downgraded, the system only uses the rank components within the current resident rank range for reconstruction. After reconstruction, the lexical order, position identifier, attention mask, and tensor shape received by the inference backend remain consistent with those when this invention is not enabled, thus allowing the continued use of the original efficient attention operators, batch processing strategies, and decoding logic.

[0060] Online factor-rank compression and local perceptual layer-by-layer factor reuse can work together. For newly entered visual lexical units, the system first determines their factor slot source based on the local candidate set and the layer-by-layer reuse mask, and writes them into the factor cache with the current resident rank of each layer as a constraint. For historical visual lexical units that have already been resident, the system releases low-contribution rank component pages based on memory pressure and candidate rank reduction priority. Therefore, the construction overhead of new visual lexical units and the resident cost of historical visual lexical units are controlled. Since both types of operations occur within the factorization cache and are uniformly reconstructed into standard key tensors and value tensors before attention calculation, the efficient backend attention operator, batching strategy, position index, and attention mask can all be implemented using the original model.

[0061] To ensure that reuse and compression decisions do not conflict, the system records the source of factor slots and the layer-side residency rank in the metadata. When a factor slot representing a lexical has been downgraded due to historical reasons and is no longer within the current residency rank range, the current visual lexical will no longer attempt to reuse that rank component, but will instead reconstruct the usable portion according to the current residency rank range. When the current visual lexical's own factors are written, only the rank components allowed by the current residency rank of each layer are written. Therefore, online reuse follows the current compression state, and online compression can uniformly apply to the physical pages corresponding to both own factors and reused factors. This collaborative mechanism avoids problems such as duplicate releases, dangling references, or inconsistent tensor shapes that arise from the complexity of factor sources.

[0062] This disclosure also provides a hierarchical factorized end-side large model KV cache reuse and compression system, including: The visual input module is used to divide visual data into multiple visual blocks; The visual embedding module is used to generate visual embeddings for generating candidate selections based on visual chunking. The factorization cache construction module is used to construct four types of visual factors: key-side head blending factor, key-side feature factor, value-side head blending factor, and value-side feature factor. The lexical management module is used to maintain a fresh, proprietary lexical pool and perform local candidate searches. An online rank compression module is used to select the rank reduction operation based on the trigger threshold, target budget, and memory loss ratio; The pagination cache management module is used to continuously maintain the current resident rank and page occupancy of each layer-side cache, and to perform page release and resident rank metadata updates. The KV reconstruction module is used to reconstruct the standard key tensor and value tensor based on the current visual factor state of each visual word, each network layer, and each key-value side, restoring the visual factors to standard KV tensors before attention computation.

[0063] Regarding the system in the above embodiments, the specific ways in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated here.

[0064] The embodiments of this application are described using an edge-side long-memory visual language assistant as an example, but the present invention is not limited to this specific application, and can also be deployed in systems that require continuous visual context, such as in-vehicle visual question answering, wearable device scene review, and edge robot environmental understanding.

[0065] This application provides a novel cache organization and management method for edge-side long-memory visual language assistants: it can adaptively compress historical caches based on factor contributions without deleting visual lexical units, and reuse truly shareable visual content layer by layer and factor slot by slot in continuous visual input, while maintaining the backend attention interface, lexical order, and tensor shape unchanged. Complex selection, deranking, and reuse decisions are confined to the internal representation layer of the cache, and the cache is restored to a standard key-value tensor at the backend computation boundary. Actual memory release is achieved through page-level management to adapt to the frequently changing resource states on mobile devices.

[0066] The effectiveness of the hierarchical factorization end-side large model KV cache reuse and compression method proposed in this application is illustrated through specific experiments below.

[0067] To verify the technical effectiveness of this invention, the Qwen3-VL-2B visual language model, the Video-MME long video understanding task, and the Jetson AGX Orin edge computing device were used as the test environment. A streaming visual chunking ingestion method was employed, and comparisons were made between the original complete KV cache under the Qwen3-VL-2B visual language model, the FastVID KV cache compression method, FactorBackbone without compression and reuse, the online factor rank compression method, and the local perceptual layer-by-layer factor reuse method. Test metrics included the overall accuracy and long video accuracy of Video-MME, visual KV resident cache, peak inference memory, visual lexical hit rate, and pre-filling latency of a single visual chunk. The test results are shown in Table 1.

[0068] Table 1 Comparison of Video-MME accuracy and edge memory usage.

[0069] As shown in Table 1, FactorBackbone maintained high overall Video-MME accuracy and long video accuracy under factorized caching space, but its resident cache still reached 1.62GB. After adopting online factor-rank compression, the resident cache was reduced to 0.81GB, and the peak memory was reduced to 6.61GB; compared with the original Qwen3-VL-2B's 2.20GB resident cache, the resident cache was reduced by approximately 63.2%, and the peak memory was reduced from 7.89GB to 6.61GB.

[0070] In continuous visual input, there is a large amount of shareable content between locally adjacent blocks. To examine the delayed benefits of the reuse mechanism, we further compared the performance of complete recalculation, whole-word reuse, and local perception layer-by-layer factor reuse under different visual similarity thresholds. The results are shown in Table 2.

[0071] Table 2 Quality-Latency Trade-off Results of Visual Cache Reuse

[0072] As shown in Table 2, while whole-word reuse can reduce chunk pre-filling latency, it significantly decreases Video-MME accuracy, indicating that directly reusing complete key-value states disrupts the current position-related attention routing. Locally perceptual layer-by-layer factor reuse, with a threshold of 0.95, reduces chunk pre-filling latency from 298.98ms (after full recalculation) to 264.74ms, achieving an acceleration of approximately 11.5%, while maintaining an overall accuracy of 58.96% and a long-video accuracy of 48.89%. When the threshold is set to 0.90, the word hit rate increases to 25.26%, the chunk pre-filling latency decreases to 250.55ms, and a relatively high overall accuracy of 57.11% is still maintained.

[0073] Compared with the prior art, the present invention has at least the following beneficial effects: First, this invention does not use direct deletion of visual lexical units as the main memory control method, but preserves the logical existence of visual lexical units and extends the queryable visual history by reducing the physical residency of low-contribution rank components.

[0074] Second, this invention performs loss-aware rank reduction based on the sensitivity differences of layers, key-value sides, and rank components, which is more suitable for edge-side memory budget fluctuation scenarios than uniformly compressing complete KV entries.

[0075] Third, the present invention reuses only the masked factor slots in continuous visual input and always uses key-side feature factors that are sensitive to the repositioning of the current lexical, thereby reducing the risk of attention routing deviation caused by reuse.

[0076] Fourth, this invention achieves page-level release through component-based master-order pagination views, avoiding large-scale cache migration during online rank reduction. Fifth, this invention still provides standard key-value tensors at the backend interface layer, facilitating compatibility with existing visual language models, efficient attention operators, and edge inference frameworks.

[0077] It should be noted that although several modules of the system for executing actions are mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided into multiple modules for embodiment. Components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of the present invention according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0078] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

[0079] It should be noted that the installation of image acquisition and personal identification equipment in public places involved in this application is necessary for maintaining public safety, complies with relevant national regulations, and is accompanied by prominent warning signs. The collected personal images and identification information can only be used for the purpose of maintaining public safety and not for other purposes; or the images, personal identification data, etc. in this application are all legally and compliantly obtained or collected with the individual's separate consent.

Claims

1. A hierarchical factorization-based end-side large model KV cache reuse and compression method, characterized in that, include: S1, Data Acquisition and Division: Acquire visual data from the edge and divide the visual data into multiple visual blocks, and divide each visual block into multiple spatial blocks; S2, Factorization Cache Unit Construction: Input all spatial blocks into the visual language model, use the visual language model to convert the spatial blocks into visual lexical units, and then perform factorization cache on the key and value of the visual lexical units so that the key and value of the visual lexical units are represented by factorization cache units. The factorization cache unit includes four types of visual factors: key-side head mixing factor, key-side feature factor, value-side head mixing factor, and value-side feature factor. S3, Offline Model Calibration: Statistically sort the contribution energy of the rank components corresponding to different network layers and key-value sides of the visual language model, and determine the upper bound of the resident rank, the lower bound of the resident rank, and the sensitivity weight of each layer's cache. S4, Online Factor Rank Compression: Real-time monitoring of the total physical residency of the visual factor cache. When the total physical residency exceeds the trigger threshold, a rank reduction operation is performed on the residency rank of the layer-side cache. S5, Local Perception Layer-by-Layer Factor Reuse: A representative word pool is preset. During the continuous input of visual words, fresh words from the visual words are stored in the representative word pool. Based on the time step and spatial block coordinates of the current visual word, candidate words are selected from the representative word pool to form a local candidate set. The cosine similarity between the current visual word and the visual embedding of each candidate word is calculated to obtain the representative word. The representative word is calibrated offline to obtain the layer-by-layer reuse mask. The layer-by-layer reuse mask is used for factor slot filling.

2. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 1, characterized in that, In S2, the key-side feature factor is injected with rotation position encoding before being written to the cache to obtain a key-side feature factor carrying the current position routing information; the key of the visual word is represented by the key-side head mixing factor and the key-side feature factor, the value of the visual word is represented by the value-side head mixing factor and the value-side feature factor, and the factorized cache unit is represented by the key-side head mixing factor, the key-side feature factor after rotation position encoding, the value-side head mixing factor, and the value-side feature factor.

3. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 2, characterized in that, The visual language model is equipped with a cache manager, and the factorization cache unit is stored in the cache manager. The cache manager is used to record the source tag of each visual word and read or reconstruct the corresponding visual factor according to the source tag. The source tag is used to distinguish whether the visual factor representing the current visual word is an own factor, a factor obtained by reusing the representative word, or a factor that no longer resides physically due to rank reduction.

4. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 1, characterized in that, In S4, a rank reduction operation is performed on the resident rank of the layer-side cache, including: The amount of memory to be released is calculated based on the number of visual lexical units, the current inhabited rank, the target inhabited rank, and the page-level storage cost corresponding to a visual lexical unit on a rank component. The representation loss is obtained based on the energy of the discarded rank component, the number of visual lexical units, and the layer-side sensitivity weights. The priority measure of candidate rank reduction operations is obtained based on the expected amount of memory released from the discarded rank components and the representation loss. Candidate de-ranking operations are sorted from smallest to largest according to priority metric, and de-ranking operations with lower priority metrics are executed first, until the cumulative release amount reaches the amount of memory to be released or all layer-side caches have reached the corresponding lower bound of the resident rank.

5. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 4, characterized in that, The target resident rank can be determined in the following ways: the intermediate value between the current resident rank and the lower bound of the resident rank is set as the target resident rank, or it can be determined adaptively based on the amount of memory to be released.

6. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 1, characterized in that, In S4, a rank reduction operation is performed on the resident rank of the layer-side cache, including: Visual factor caches are addressed according to network layer, key value side, rank component index and lexical page. A group of visual factors with the same layer side and the same rank component are organized into page-level management units. For the key-side page, the page includes the key-side header mixing factor corresponding to the rank component and the key-side feature factor after rotation position encoding; For the value-side page, the page includes the value-side header mixing factor and the value-side feature factor corresponding to the rank component; When the residency rank of the target layer cache is reduced to the target residency rank, pages with rank indexes greater than the target residency rank are released directly, and the residency rank metadata of that layer is updated.

7. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 1, characterized in that, In S5, if the local candidate set is empty, or the highest cosine similarity is lower than the preset reuse threshold, the current visual word element is represented by the reconstructed visual factors, and after the current visual block processing is completed, the current visual word element is placed into the representative word element pool as a fresh free word element.

8. The hierarchical factorization end-side large model KV cache reuse and compression method according to claim 1, characterized in that, In S5, the layer-by-layer multiplexing mask includes a first layer set and a second layer set. The first layer set is used to control whether the key-side head mixing factor and the value-side head mixing factor can be reused; the second layer set is used to control whether the value-side feature factor can be reused.

9. The hierarchical factorization end-side large model KV cache reuse and compression method according to any one of claims 1 to 8, characterized in that, The collaborative process of online factor rank compression and local sensing layer-by-layer factor reuse includes: For visual lexical units newly entering the visual language model, their factor slot sources are determined based on the local candidate set and the layer-by-layer reuse mask, and they are written into the factor cache with the current side-resident rank of each layer as a constraint. For historical visual terms that have already been resident, release the pages of low-contribution-rank components based on memory pressure and the priority of candidate de-ranking operations.

10. A hierarchical factorized end-side large model KV cache reuse and compression system, characterized in that, include: The visual input module is used to divide visual data into multiple visual blocks; The visual embedding module is used to generate visual embeddings for generating candidate selections based on visual chunking. The factorization cache construction module is used to construct four types of visual factors: key-side head blending factor, key-side feature factor, value-side head blending factor, and value-side feature factor. The lexical management module is used to maintain a fresh, proprietary lexical pool and perform local candidate searches. An online rank compression module is used to select the rank reduction operation based on the trigger threshold, target budget, and memory loss ratio; The pagination cache management module is used to perform page release and resident rank metadata updates; The KV reconstruction module is used to restore visual factors to standard KV tensors before attention calculation.

Images