Weight and cache management system and method for a model

Abstract

This invention provides a model weight and cache management system and method, belonging to the field of deep learning model deployment technology. The model weight and cache management system of this invention performs multi-level compression through an intelligent weight compression module, and greatly reduces storage costs and network transmission bandwidth pressure through computation graph perception and access pattern prediction, shortening startup waiting time. The differential version management engine makes daily model fine-tuning and updates extremely lightweight, and the hardware adaptive optimizer enables the same compressed weights to dynamically adapt to different hardware architectures at runtime, significantly improving computational efficiency and cache hit rate.

Description

Technical Field

[0001] This invention relates to the field of deep learning model deployment technology, and in particular to a model weight and caching management system and method. Background Technology

[0002] With the rapid development of artificial intelligence technology, the parameter scale of deep learning models, especially large language models based on self-attention mechanisms, has grown exponentially, rapidly expanding from millions of parameters in the early stages to hundreds of billions or even trillions of parameters. This explosive growth in parameter scale has brought unprecedented challenges to the deployment, storage, distribution, and inference operation of models, forming memory and bandwidth bottlenecks.

[0003] In existing technologies, model weight management typically employs a relatively primitive file system storage method. That is, the trained model parameters are stored as large binary files on hard drives or cloud storage. When model inference is required, the computing system needs to load the entire model file from the disk into main memory, and then transfer it from main memory to the GPU or other accelerator's video memory. This traditional management method has significant technical drawbacks. Existing general data compression techniques (such as LZ77 and Huffman coding) are not optimized for the numerical distribution characteristics of neural network weight data, resulting in limited compression ratios. The huge storage footprint of large models not only increases hardware costs but also leads to extremely high latency during network distribution and disk reads, severely impacting the real-time responsiveness of services. During model inference, not all weight parameters are used simultaneously. However, traditional loading mechanisms often employ a full-load strategy, loading all weights into video memory at once. This not only leads to a significant waste of video memory resources, limiting the scale of models that can be deployed on a single GPU, but also results in extremely long startup times. Furthermore, during continuous model iteration and fine-tuning, typically only a small portion of parameters undergo significant changes. However, the existing management system lacks fine-grained version control capabilities, and each fine-tuning requires saving a complete copy of the model, resulting in an exponential increase in storage space.

[0004] Therefore, there is an urgent need to develop a new model weight and caching management system to address the aforementioned industry pain points. Summary of the Invention

[0005] This invention provides a model weight and cache management system and method to address the shortcomings of high deployment costs and low deployment efficiency in the prior art, thereby reducing storage costs and improving computational efficiency.

[0006] This invention provides a weight and cache management system, including: an intelligent weight compression module, a predictive cache management module, a differential version management engine, and a hardware adaptive optimizer; The intelligent weight compression module is used to perform multi-level compression on the weight data; The predictive cache management module includes an access pattern predictor, which is used to parse the computation graph structure and inter-layer dependencies of the currently running model in real time, and predict the weighted blocks that will be called in the future time window by combining historical access frequency characteristics. The predictive cache management module performs intelligent prefetching operation based on the prediction results, and dynamically increases the cache block size when a sequential access pattern is detected, and decreases the cache block granularity when a random access pattern is detected, so as to adjust the dynamic caching strategy. The differential version management engine is used to generate incremental data based on the numerical differences between the weight matrices of the old and new versions, and to associate and store the incremental data with the base version identifier. When a loading instruction is received, the target weight is reconstructed based on the base version and the incremental data. The hardware adaptive optimizer is used during the weight loading phase to read the hardware configuration description file of the target deployment environment, and automatically adjust the memory layout and numerical precision of the decompressed weights according to the vector register width and cache line size of the hardware, generating a hardware-specific optimized version for inference.

[0007] According to the model weight and cache management system provided by the present invention, the intelligent weight compression module is further used to perform structural compression on the weight data through sparsity detection and singular value decomposition to remove redundant parameters, perform adaptive quantization encoding on the structurally compressed data based on the numerical distribution characteristics, and perform entropy encoding on the quantized data to generate compressed weight packages and corresponding decompressed metadata.

[0008] According to the model weight and cache management system provided by the present invention, the intelligent weight compression module is further used to detect the sparsity of weight data when performing structural compression; The intelligent weight compression module stores non-zero elements using compressed sparse rows or compressed sparse columns when the sparsity is higher than a preset threshold. When the weight data has low-rank characteristics, the intelligent weight compression module uses singular value decomposition to decompose the weight data into the product of two low-dimensional matrices for storage. The intelligent weight compression module is also used to employ a block compression strategy when the weight data does not satisfy the condition of having a sparsity higher than the preset threshold and having low-rank characteristics, dividing the weight data into multiple sub-blocks for separate processing.

[0009] According to the present invention, a model weight and cache management system is provided, wherein the input features of the predictor include the topological position of the current model layer, the dimension information of the data flow tensor, and the weight block access sequence of the past N time steps; the output feature of the predictor is the probability value of each weight block being accessed in the next time window.

[0010] According to the model weight and cache management system provided by the present invention, the predictive cache management module is further configured to mark the target weight block as resident in memory when the access frequency of the target weight block is detected to be greater than the hot spot threshold; the weight block in resident in memory is not replaced by the cache eviction algorithm until the inference session of the model ends. According to the weight and cache management system of the model provided by the present invention, a distributed collaborative management module is further included. The distributed collaborative management module is used to connect multiple computing nodes. The distributed collaborative management module is used to analyze the cache state graph of each node, formulate a synchronization plan that minimizes network transmission, and send compressed incremental data packets to nodes that already hold the baseline version when distributing new weights.

[0011] According to the weight and cache management system of the model provided by the present invention, the distributed collaborative management module further includes a global cache mapping table. The global cache mapping table is used to record the weight version ID and block index stored in all connected nodes in real time, so as to request weight data from the nearest node with the lowest network latency when a cache miss occurs on the local node.

[0012] The present invention also provides a method for managing the weights and cache of a model, applied to a weight and cache management system for a model as described in any of the preceding claims, comprising: The system receives the model's weight data, performs multi-level compression through structural analysis, adaptive quantization, and entropy coding, generates compressed weights and metadata, and stores them in the storage medium. In response to the model inference request, the trained predictor is used to predict the weight blocks required for the next time step based on the computation graph structure; In a background thread, the corresponding compressed weight block is pre-fetched based on the prediction result and then decompressed in memory. Obtain the hardware characteristics of the current deployment environment, and rearrange the memory layout and precision of the decompressed weights; The processed weights are provided to the inference engine, and the cache replacement strategy is dynamically adjusted based on real-time access patterns.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements a weight and cache management method as described in any of the above-described models.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a weight and cache management method for any of the models described above.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a weight and cache management method for any of the models described above.

[0016] The model weight and cache management system and method provided by this invention perform multi-level compression through an intelligent weight compression module, and greatly reduce storage costs and network transmission bandwidth pressure through computation graph perception and access pattern prediction, shortening startup waiting time. The differential version management engine makes daily model fine-tuning and updates extremely lightweight, and the hardware adaptive optimizer enables the same compressed weight to dynamically adapt to different hardware architectures at runtime, significantly improving computational efficiency and cache hit rate. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the weight and cache management system structure of the model provided by the present invention; Figure 2 This is a flowchart illustrating the weight and cache management method of the model provided by the present invention; Figure 3 is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0020] The following is combined Figure 1 Figure 3 illustrates the weight and cache management system and method of the model of the present invention.

[0021] The system architecture of this invention can logically reside between the deep learning inference framework and the underlying operating system / hardware driver. For example... Figure 1As shown, the model's weight and cache management system of this invention mainly consists of four functional units: an intelligent weight compression module, a predictive cache management module, a differential version management engine, and a hardware adaptive optimizer. Furthermore, for distributed deployment scenarios, the model's weight and cache management system may also include a distributed collaborative management module.

[0022] The intelligent weight compression module is used to perform multi-level compression on weighted data. It employs a serial pipeline processing architecture, with data sequentially passing through a structured analysis layer, a numerical quantization layer, and an entropy coding layer.

[0023] Structural Analysis Layer: When the system receives the matrix of weight data to be processed, it first activates the sparsity detection unit. This unit traverses the entire weight matrix, calculating the proportion of non-zero elements in the total elements, i.e., the sparsity. The system internally presets a sparsity threshold, such as 60% or 80%. If the detected sparsity is higher than the preset threshold, the system determines that the matrix is ​​a sparse matrix. At this point, the module abandons the traditional two-dimensional array storage method and instead adopts a compressed sparse row or compressed sparse column format. Specifically, taking the compressed sparse row format as an example, the system can create three one-dimensional arrays: a value array, a column index array, and a row offset array. The value array is used to store the values ​​of all non-zero elements, the column index array is used to store the column number corresponding to each non-zero element, and the row offset array is used to store the starting position of the first non-zero element in each row in the value array. This conversion can greatly reduce the storage space occupied by zero elements.

[0024] If the sparsity is below a threshold, the system will further initiate singular value decomposition or principal component analysis algorithms to detect the rank of the matrix. If the weight matrix is ​​found to have low-rank characteristics, meaning that most of the information in the matrix is ​​concentrated in a few singular values, the system can perform matrix decomposition. For example, a large matrix of dimension M×N can be approximately decomposed into the product of two smaller matrices: an M×K left singular matrix and a K×N right singular matrix, where K is much smaller than M and N. The system can store only these two smaller matrices, thereby reducing the number of parameters from M×N to K×(M+N). For dense and full-rank weights that do not satisfy either high sparsity or low-rank characteristics, such as in the convolutional kernel of a convolutional layer, the system can adopt a block-based strategy. The large matrix is ​​cut into small sub-blocks, such as 32×32 or 64×64, and each sub-block is quantized independently. This approach can fully utilize the numerical correlation of local data, because the weight values ​​of the same layer in the neural network have similar dynamic ranges in a local range.

[0025] Understandably, in some embodiments, the intelligent weight compression module is also used to detect the sparsity of the weight data during structural compression; when the sparsity is higher than a preset threshold, the intelligent weight compression module uses compressed sparse rows or compressed sparse columns to store non-zero elements; when the weight data has low-rank characteristics, the intelligent weight compression module uses singular value decomposition to decompose the weight data into the product of two low-dimensional matrices for storage; the intelligent weight compression module is also used to use a block compression strategy to divide the weight data into multiple sub-blocks for processing when the weight data does not satisfy the condition that the sparsity is higher than the preset threshold and has low-rank characteristics.

[0026] Numerical Quantization Layer: Data that has undergone structural compression or raw block data enters the quantization stage. The intelligent weighted compression module does not use globally uniform quantization parameters, but instead employs adaptive statistical quantization.

[0027] The intelligent weighted compression module can statistically analyze the numerical histogram of the current data block to determine whether it conforms to a Gaussian, Laplace, or uniform distribution. Based on the variance and extrema of the numerical distribution, the system can automatically determine the quantization bit width, such as 8 bits, 4 bits, or even as low as 2 bits. For data with high numerical concentration, fewer bits are used; for data with many outliers, a non-linear quantization codebook is used.

[0028] To maintain accuracy, the intelligent weighted compression module calculates and stores the quantization scaling factor and zero-point offset for each data block. This metadata is packaged together with the quantized integer data.

[0029] Entropy coding layer: Although the bit width of the integer stream after entropy coding quantization is reduced, statistical redundancy still exists. Entropy coding algorithms such as arithmetic coding or asymmetric number systems can be used to perform lossless compression of the data stream. Entropy coding utilizes the probability differences of data occurrences, mapping high-frequency numerical sequences to short codewords and low-frequency sequences to long codewords, further approaching the Shannon limit.

[0030] While performing all the above operations, the intelligent weight compression module can simultaneously generate a detailed decompression metadata file. The metadata records the compression method, quantization parameters, matrix dimensions, and byte offset of each weight layer in the compressed package. This metadata can serve as a key index for subsequent fast reading.

[0031] It is understood that, in the above embodiments, the intelligent weight compression module is also used to perform structural compression on the weight data through sparsity detection and singular value decomposition to remove redundant parameters, perform adaptive quantization encoding on the structurally compressed data based on the numerical distribution characteristics, and perform entropy encoding on the quantized data to generate compressed weight packages and corresponding decompressed metadata.

[0032] The predictive cache management module includes an access pattern predictor. The predictor is used to parse the computation graph structure and inter-layer dependencies of the currently running model in real time, and combined with historical access frequency characteristics, predict the weighted blocks that will be called in the future time window. The predictive cache management module performs intelligent prefetching operations based on the prediction results, and dynamically increases the cache block size when a sequential access pattern is detected, and decreases the cache block granularity when a random access pattern is detected, so as to adjust the dynamic caching strategy.

[0033] When the inference engine loads the model, the predictive cache management module reads the model's computation graph definition file, such as ONNX format or a framework-specific graph structure. The computation graph is a directed acyclic graph, where nodes represent operators such as matrix multiplication, convolution, and activation functions, and edges represent data flow dependencies.

[0034] The predictive cache management module can perform topological sorting of the computation graph to determine the execution order of all operators. Static topology provides a basic order, but in models containing branches, loops, or dynamic input lengths, the execution path is dynamic.

[0035] The predictor's input features include the current model layer's topological position, the data flow tensor's dimensionality, and the weight block access sequence over the past N time steps. The predictor's output feature is the probability value of each weight block being accessed in the next time window. The predictor receives multi-dimensional feature input, including the currently executing operator ID, the tensor dimension of the current input data, and the actual weight block access sequence over the past N time steps. Internally, the predictor can run a lightweight Markov chain model or a recurrent neural network model, outputting the probability values ​​of each weight block being accessed in the next time window in real time.

[0036] The predictive cache management module can perform prefetching based on prediction results by launching a background direct memory access thread or a separate I / O thread. A safe prefetch time window can be calculated based on the current hardware's I / O bandwidth and decompression speed. For example, if loading and decompressing a weight block takes 10 milliseconds, a load instruction must be issued at least 10 milliseconds before the computation unit actually needs the data. For weight blocks with a predicted probability higher than a set threshold, they can be immediately read from disk or network storage and decompressed into the cache. For branch weights with a low probability but still potentially accessible, partial prefetching can be performed—that is, only the compressed data is read but not immediately decompressed, or a low-precision preview version is loaded.

[0037] When the predictor detects that the model is performing large-scale continuous matrix operations, such as calculations in a fully connected layer, and the access address exhibits a strictly linear growth, it can be identified as a sequential access pattern. In this case, the cache block size is dynamically increased, for example, from 4KB to 2MB or even larger. This large-granularity caching reduces the overhead of metadata management, fully utilizes the high bandwidth characteristics of sequential reads on storage devices, and significantly improves throughput. However, when the model executes sparse attention mechanisms or expert hybrid model routing, the access pattern exhibits high randomness. In this case, continuing to use large cache blocks will lead to severe cache pollution, reading in large amounts of useless data. Upon detecting this pattern, the cache block granularity can be immediately reduced, for example, adjusted to 1KB or smaller sub-blocks, precisely loading only the activated specific parameter blocks, thereby improving effective bandwidth utilization.

[0038] In some embodiments, the predictive cache management module is further configured to mark the target weight block as resident in memory when the access frequency of the target weight block is detected to be greater than the hot spot threshold; the weight block in resident memory is not replaced by the cache eviction algorithm until the inference session of the model ends.

[0039] The predictive cache management module can also monitor global access frequency. Once the access frequency of a weighted block exceeds the hotspot threshold, its status can be marked as resident in memory. During subsequent cache eviction processes, i.e., when cache space is insufficient and old data needs to be released, blocks marked as resident can be exempted from removal by the replacement algorithm. This locked state persists until the current inference session completely ends, ensuring zero-latency access on the critical path.

[0040] The differential version management engine is used to generate incremental data based on the numerical differences between the weight matrices of the old and new versions, and to associate and store the incremental data with the base version identifier. When a loading instruction is received, the target weights are reconstructed based on the base version and the incremental data.

[0041] When a new version of the model weights is received, such as a fine-tuned version B, version B is not stored directly. The difference version management engine can first lock an older version, such as version A, as a baseline. Then, the weight matrices of version B and version A are compared element-wise. During model fine-tuning, usually only some parameters change, or most parameters change very little. The difference version management engine can calculate the difference, apply a threshold filter to the difference matrix, and treat elements with absolute values ​​less than a certain tolerance as zero, making the difference matrix a highly sparse matrix.

[0042] It is possible to store only the compressed sparse difference matrix and establish a metadata link pointing to a unique identifier of the baseline version A. In this case, the space required to store the new version will be significantly reduced.

[0043] When an inference request specifies loading version B, the system first reads version B's metadata and discovers that it depends on the baseline version A. It can then read the weights of version A from storage or cache. Further, the system can acquire incremental data, decompress it in memory, and perform vector addition operations.

[0044] To further improve efficiency, if the hardware supports it, addition operations can be performed dynamically as the weights are loaded into the registers, without the need to generate intermediate copies in main memory.

[0045] The hardware adaptive optimizer is used during the weight loading phase to read the hardware configuration description file of the target deployment environment, and automatically adjust the memory layout and numerical precision of the decompressed weights according to the vector register width and cache line size of the hardware, generating a hardware-specific optimized version for inference.

[0046] The hardware adaptive optimizer ensures that data exists in memory in the form most suitable for the target computing unit, eliminating runtime format conversion overhead.

[0047] The hardware configuration description file can be read during startup or deployment. This file defines in detail the microarchitectural parameters of the target hardware. These parameters may include vector register width, cache line size, and tensor core layout requirements.

[0048] Based on the above parameters, the hardware adaptive optimizer can rearrange the physical addresses of the decompressed weights. To prevent performance penalties caused by unaligned memory accesses, the hardware adaptive optimizer can also add padding data at the end of each line to ensure that the starting address of each line is divisible by the cache line size. If the hardware prefers channel priority and the original weights are at the end of the channel, the hardware adaptive optimizer can automatically perform a transpose operation during the loading phase.

[0049] In some embodiments, to improve cache hit rate, the hardware adaptive optimizer can reorganize large matrices in memory into a sequence of micro-blocks arranged in a zigzag or Hilbert curve pattern, so that the associated data is stored contiguously in physical memory when performing convolution window sliding or matrix block multiplication.

[0050] The hardware adaptive optimizer can also dynamically adjust the numerical type during decompression based on the hardware's precision support. If the target hardware supports low-precision inference and the user allows for a slight loss of precision, 32-bit floating-point numbers can be directly truncated or rounded to 16-bit floating-point numbers, thereby halving memory bandwidth usage and utilizing the hardware's low-precision acceleration instructions.

[0051] The model weight and cache management system provided in this embodiment of the invention performs multi-level compression through an intelligent weight compression module, and greatly reduces storage costs and network transmission bandwidth pressure through computation graph perception and access pattern prediction, shortens startup waiting time, and makes daily model fine-tuning and updates extremely lightweight through a differential version management engine. The hardware adaptive optimizer enables the same compressed weight to dynamically adapt to different hardware architectures at runtime, which greatly improves computational efficiency and cache hit rate.

[0052] In some embodiments, the weight and cache management system of the model in this embodiment of the invention further includes a distributed collaborative management module, which is used to connect multiple computing nodes. The distributed collaborative management module is used to analyze the cache state graph of each node, formulate a synchronization plan that minimizes network transmission, and send compressed incremental data packets to nodes that already hold the baseline version when distributing new weights.

[0053] In multi-card or multi-machine cluster environments, the distributed collaborative management module is used to implement weight sharing and communication optimization mechanisms.

[0054] The distributed collaborative management module also includes a global cache mapping table, which is used to record the weight version ID and block index stored in all connected nodes in real time, so as to request weight data from the nearest node with the lowest network latency when a cache miss occurs on the local node.

[0055] In some embodiments, a global cache mapping table can be maintained on the cluster's management node or in a decentralized manner. The global cache mapping table records in real time which versions of weight blocks reside in the GPU memory and RAM of each compute node in the cluster, as well as the specific block index range.

[0056] It's important to note that each node can periodically send heartbeat packets, reporting a summary of its cached content. When a compute node experiences a cache miss, it can avoid requesting data from a remote file server. Instead, it first queries the global mapping table to find other nodes holding the target weight block. The network latency and bandwidth cost of transferring data from each holding node to this node can be calculated. Typically, the transmission speed between nodes within the same rack is much higher than that across racks or data centers. The system can select the nearest node with the lowest network latency to establish a high-speed point-to-point connection, such as remote direct memory access, directly copying data from node B's memory to node A.

[0057] When distributing new copyrighted data, the distributed collaborative management module can calculate the minimum spanning tree or use a BitTorrent-like protocol to avoid all nodes downloading from the source server simultaneously. Instead, it first distributes the data to a few seed nodes and then uses the high-speed interconnection network within the cluster for fission propagation, greatly shortening the overall model update time of the cluster.

[0058] The following describes the weight and cache management method of the model provided by the present invention. The weight and cache management method of the model described below can be referred to in correspondence with the weight and cache management system of the model described above.

[0059] like Figure 2 As shown, the model weight and cache management method of this embodiment is applied to the model weight and cache management system in the foregoing embodiment. The model weight and cache management method of this embodiment includes steps 210, 220, 230, 240 and 250.

[0060] Step 210: Receive the model's weight data, perform multi-level compression through structural analysis, adaptive quantization, and entropy coding, generate compressed weights and metadata, and store them in the storage medium. Step 220: In response to the model inference request, use the trained predictor to predict the weight blocks required for the next time step based on the computation graph structure; Step 230: In the background thread, the corresponding compressed weight block is pre-fetched according to the prediction result and decompressed in memory; Step 240: Obtain the hardware characteristics of the current deployment environment, and rearrange the memory layout and precision of the decompressed weights. Step 250: The processed weights are provided to the inference engine, and the cache replacement strategy is dynamically adjusted according to the real-time access patterns.

[0061] It is understood that the steps of this method can be referred to the description of the model weights and cache management system modules in the foregoing embodiments.

[0062] Specifically, the method may include the following procedures.

[0063] Users submit the original model weights. The intelligent weight compression module first performs structural analysis. If a layer's weight sparsity reaches 90%, it is converted to a compressed sparse row format; if a layer is dense but low-rank, singular value decomposition is performed. Adaptive quantization is then performed to generate a quantization index and codebook. Finally, entropy encoding is performed to generate the final compressed weight package and metadata, which is written to persistent storage. For incremental updates, the difference version management engine can calculate the difference from the baseline version and only store the difference package.

[0064] After the inference engine starts, the hardware adaptive optimizer reads the hardware descriptor of the current machine. For example, if it detects that it is a certain type of tensor processor with a vector width of 512 bits, the predictive cache management module can load the model computation graph, establish the dependency topology, and initialize the probabilistic prediction model.

[0065] During the execution of layer T computation, the predictor can calculate the set of high-probability accesses for layer T+k weight blocks based on the current state. A background thread can check the cache. If the target block is not in the cache, a prefetch request is initiated. If an incremental version is requested, the baseline block and incremental block are read from storage and merged for reconstruction. If a distributed missing data occurs, it is pulled from neighboring nodes. After data enters memory, it can be immediately decompressed, and memory alignment and layout rearrangement are performed when writing to the target buffer according to the instructions of the hardware adaptive optimizer, for example, rearranging the data into 8×32 micro-blocks. The monitoring thread can also analyze the access flow in real time. If continuous reads are detected, the memory allocator is notified to allocate large page memory blocks; if hot weights such as commonly used word embedding vectors are detected, they can be locked in the cache and not released.

[0066] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3 As shown, the electronic device may include a processor 310, a communication interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communication interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute a model weight and cache management method. This method includes: receiving the model's weight data, performing multi-level compression through structural analysis, adaptive quantization, and entropy encoding to generate compressed weights and metadata and storing them in a storage medium; responding to a model inference request, using a trained predictor to predict the weight blocks required for the next time step based on the computational graph structure; pre-fetching the corresponding compressed weight blocks in a background thread based on the prediction results and decompressing them in memory; acquiring the hardware characteristics of the current deployment environment, rearranging the memory layout and adapting the precision of the decompressed weights; providing the processed weights to the inference engine, and dynamically adjusting the cache replacement strategy according to the real-time access pattern.

[0067] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0068] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the weight and cache management methods of the model provided by the above methods. The method includes: receiving the model's weight data, performing multi-level compression through structural analysis, adaptive quantization, and entropy coding to generate compressed weights and metadata and storing them in the storage medium; responding to a model inference request, using a trained predictor to predict the weight blocks required for the next time step based on the computation graph structure; pre-fetching the corresponding compressed weight blocks in a background thread according to the prediction results and decompressing them in memory; obtaining the hardware characteristics of the current deployment environment, rearranging the memory layout and adapting the precision of the decompressed weights; providing the processed weights to the inference engine, and dynamically adjusting the cache replacement strategy according to the real-time access mode.

[0069] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a weight and cache management method for the model provided by the above-described methods. The method includes: receiving the model's weight data; performing multi-level compression through structural analysis, adaptive quantization, and entropy coding to generate compressed weights and metadata and storing them in the storage medium; responding to a model inference request, using a trained predictor to predict the weight blocks required for the next time step based on the computational graph structure; pre-fetching the corresponding compressed weight blocks in a background thread according to the prediction results and decompressing them in memory; acquiring the hardware characteristics of the current deployment environment, rearranging the memory layout and adapting the precision of the decompressed weights; providing the processed weights to the inference engine, and dynamically adjusting the cache replacement strategy according to the real-time access pattern.

[0070] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0071] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0072] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A weight and cache management system for a model, characterized in that, include: The system includes an intelligent weighted compression module, a predictive cache management module, a differential version management engine, and a hardware adaptive optimizer. The intelligent weight compression module is used for multi-level compression of weight data; The predictive cache management module includes an access pattern predictor, which is used to parse the computation graph structure and inter-layer dependencies of the currently running model in real time, and predict the weighted blocks that will be called in the future time window by combining historical access frequency characteristics. The predictive cache management module performs intelligent prefetching operation based on the prediction results, and dynamically increases the cache block size when a sequential access pattern is detected, and decreases the cache block granularity when a random access pattern is detected, so as to adjust the dynamic caching strategy. The differential version management engine is used to generate incremental data based on the numerical differences between the weight matrices of the old and new versions, and to associate and store the incremental data with the base version identifier. When a loading instruction is received, the target weight is reconstructed based on the base version and the incremental data. The hardware adaptive optimizer is used during the weight loading phase to read the hardware configuration description file of the target deployment environment, and automatically adjust the memory layout and numerical precision of the decompressed weights according to the vector register width and cache line size of the hardware, generating a hardware-specific optimized version for inference.

2. The weight and cache management system for the model according to claim 1, characterized in that, The intelligent weight compression module is also used to perform structural compression on the weight data through sparsity detection and singular value decomposition to remove redundant parameters, perform adaptive quantization encoding on the structurally compressed data based on the numerical distribution characteristics, and perform entropy encoding on the quantized data to generate compressed weight packages and corresponding decompressed metadata.

3. The weight and cache management system for the model according to claim 2, characterized in that, The intelligent weight compression module is also used to detect the sparsity of weight data when performing structural compression; The intelligent weighted compression module stores non-zero elements in a compressed sparse row or compressed sparse column format when the sparsity is higher than a preset threshold. The intelligent weight compression module uses singular value decomposition to decompose the weight data into the product of two low-dimensional matrices for storage when the weight data has low-rank characteristics. The intelligent weight compression module is also used to use a block compression strategy to divide the weight data into multiple sub-blocks for processing when the weight data does not meet the condition that the sparsity is higher than a preset threshold and has low-rank characteristics.

4. The weight and cache management system for the model according to claim 1, characterized in that, The input features of the predictor include the topological position of the current model layer, the dimensional information of the data flow tensor, and the access sequence of the weight blocks over the past N time steps; the output feature of the predictor is the probability value of each weight block being accessed in the next time window.

5. The weight and cache management system for the model according to claim 1, characterized in that, The predictive cache management module is also used to mark the target weight block as resident in memory when the access frequency of the target weight block is detected to be greater than the hot spot threshold; the weight block in resident memory is not replaced by the cache eviction algorithm until the inference session of the model ends.

6. The weight and cache management system for the model according to claim 1, characterized in that, It also includes a distributed collaborative management module, which is used to connect multiple computing nodes. The distributed collaborative management module is used to analyze the cache state graph of each node, formulate a synchronization plan that minimizes network transmission, and send compressed incremental data packets to nodes that already hold the baseline version when distributing new copyright weights.

7. The weight and cache management system for the model according to claim 6, characterized in that, The distributed collaborative management module also includes a global cache mapping table, which is used to record the weight version ID and block index stored in all connected nodes in real time, so as to request weight data from the nearest node with the lowest network latency when a cache miss occurs on the local node.

8. A method for managing the weights and cache of a model, applied to a weight and cache management system for the model according to any one of claims 1-7, the method comprising: The system receives the model's weight data, performs multi-level compression through structural analysis, adaptive quantization, and entropy coding, generates compressed weights and metadata, and stores them in the storage medium. In response to the model inference request, the trained predictor is used to predict the weight blocks required for the next time step based on the computation graph structure; In a background thread, the corresponding compressed weight block is pre-fetched based on the prediction result and then decompressed in memory. Obtain the hardware characteristics of the current deployment environment, and rearrange the memory layout and precision of the decompressed weights; The processed weights are provided to the inference engine, and the cache replacement strategy is dynamically adjusted based on real-time access patterns.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it implements the weight and cache management method of the model as described in claim 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the weight and cache management method of the model as described in claim 8.

Images