Method, system, and computer program product for sparse quantization of a model

Abstract

The present application discloses methods, systems and computer program products for sparse quantization of a model. The method comprises: generating a fine-tuning dataset based on data associated with a target task of a pre-trained model; performing multiple rounds of progressive sparsification on the pre-trained model; and performing multiple rounds of progressive quantization on the model after the multiple rounds of progressive sparsification. Each round of progressive sparsification comprises: performing a sparse operation; performing hierarchical fine-tuning on the model resulting from the sparse operation using the fine-tuning dataset, and imposing a sparsity structure constraint during the hierarchical fine-tuning. Each round of progressive quantization comprises: performing a quantization operation; performing hierarchical fine-tuning on the model resulting from the quantization operation using the fine-tuning dataset, and imposing both a sparsity structure constraint and a quantization structure constraint during the hierarchical fine-tuning. The method achieves effective maintenance of the performance of the model on the target task while significantly reducing the model storage footprint and computational complexity.

Description

Technical Field

[0001] This application generally relates to the field of artificial intelligence, and more specifically to a method, system, and computer program product for sparse quantization of models. Background Technology

[0002] With the rapid development of artificial intelligence technology, sparse quantization has become a mainstream optimization method to reduce the deployment cost of large models and improve inference efficiency. However, existing sparse quantization techniques for large models generally suffer from the core problem of "accuracy-efficiency" imbalance. In particular, sparse quantization is prone to model mode collapse and significant degradation of task performance, which seriously restricts the practical application of the technology. Summary of the Invention

[0003] To address the aforementioned problems in the prior art, this application proposes a method for sparse quantization of a model, as well as a system, computer program product, and non-transitory computer-readable medium associated with the method.

[0004] According to a first aspect of this application, a method for sparse quantization of a model is provided. The method includes: generating a fine-tuning dataset based on data associated with a target task of a pre-trained model; performing multiple rounds of progressive sparsity processing on the pre-trained model; and performing multiple rounds of progressive quantization processing on the model after the multiple rounds of progressive sparsity processing. Each round of progressive sparsity processing includes: performing a sparsity operation; performing hierarchical fine-tuning of the model obtained after the sparsity operation using the fine-tuning dataset; and applying sparse structure constraints during the hierarchical fine-tuning process. Each round of progressive quantization processing includes: performing a quantization operation; performing hierarchical fine-tuning of the model obtained after the quantization operation using the fine-tuning dataset; and applying both sparse structure constraints and quantization structure constraints during the hierarchical fine-tuning process.

[0005] According to a second aspect of this application, a system for sparse quantization of a model is provided. The system includes a processor and a memory. The memory stores instructions that, when executed by the processor, perform the methods described in this application.

[0006] In other aspects of this application, a non-transitory computer-readable medium storing instructions and a computer program product including the instructions are provided. These instructions, when executed by one or more processors, cause the processors to perform the methods described in this application. Attached Figure Description

[0007] The operation and function of these and other features disclosed herein, as well as related structural elements and combinations thereof, will become more apparent upon consideration of the following description and appended claims with reference to the accompanying drawings (all of which form a part of this specification, wherein like reference numerals denote corresponding portions in the drawings). However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and not for limiting purposes. In the drawings:

[0008] Figure 1 A flowchart of a method for sparse quantization of a model according to an embodiment of this application is shown.

[0009] Figure 2 An example flowchart of multi-round progressive sparsity processing according to an embodiment of this application is shown.

[0010] Figure 3 An example flowchart of multi-round progressive quantization processing according to an embodiment of this application is shown.

[0011] Figure 4 This application illustrates an implementation method based on an embodiment of the present application. Figure 1 An exemplary system diagram of the neural network (NN) processing unit of the method shown.

[0012] Figure 5 An example computing device in which a processor core according to an embodiment of the present disclosure may be used is shown. Detailed Implementation

[0013] The following disclosure provides numerous different embodiments or examples for implementing various features of the provided subject matter. Specific examples of components and arrangements are described below to simplify this application. Of course, these are merely examples and not limiting.

[0014] In fields such as large language models and computer vision models in artificial intelligence, the scale of model parameters continues to expand to the billions or even trillions, placing extremely high demands on hardware storage resources and computing power. To reduce the deployment cost of large models and improve inference efficiency, sparse quantization technology has become a mainstream optimization method. By sparsely processing model weights (removing redundant parameters) and quantizing them (reducing numerical precision, such as mapping to 8 bits or 4 bits), sparse quantization technology can significantly reduce memory usage and computational complexity while maintaining the basic functionality of the model. Therefore, it has been widely explored in scenarios such as edge device deployment and large-scale cloud inference.

[0015] However, existing sparse quantization schemes generally face the challenge of balancing accuracy and efficiency in practice. For example, some techniques employ a "sparse quantization without fine-tuning" approach. This approach, after performing sparse and quantized processing on the model weights, directly uses the resulting lightweight model for inference tasks without introducing subsequent fine-tuning or adjustment. The advantage of this approach is its simplicity and ability to fully retain the memory reduction and computational acceleration effects brought about by sparse and quantized operations. However, such a sparse and quantized process inevitably damages the model's original parameter distribution and feature representation capabilities, easily leading to "pattern collapse" and loss of the ability to distinguish complex semantic or visual features. For example, in text generation tasks using large language models, after processing with this approach, the model may continuously output repetitive sentences, failing to generate logically coherent content.

[0016] Some techniques employ a "full-data re-tuning scheme." This scheme aims to compensate for the accuracy loss caused by sparse quantization by retraining the compressed model using all of its original training data (i.e., re-tuning). This scheme can usually restore the model's performance to some extent. However, this scheme is extremely computationally expensive, requiring a significant amount of computing resources and time for full-data fine-tuning; furthermore, full-data fine-tuning is prone to overfitting, meaning the model overfits to the specific distribution of the training data, resulting in poor generalization ability on unseen test data.

[0017] Some techniques employ a compromise: a "globally unified lightweight fine-tuning scheme." This scheme uses a small amount of data (lightweight) to fine-tune the sparsely quantized model to control computational costs. It uses a globally unified fine-tuning strategy, adjusting all network layers of the model using the same learning rate, number of iterations, and other hyperparameters. However, because different layers in a large model have different functional roles (e.g., lower layers handle basic feature extraction, upper layers handle advanced semantic fusion), the degree of damage caused by sparse quantization varies significantly across layers. Globally unified fine-tuning cannot accurately correct parameter defects in individual layers and still struggles to completely solve the mode collapse problem.

[0018] Some techniques employ a "fine-tuning process that disrupts sparse structure" approach. This approach allows all weight parameters of the model to be updated during fine-tuning without constraining the model's sparse structure. While this unconstrained fine-tuning may help the model adapt more freely to the data to restore accuracy, it can severely disrupt the obtained compressed structure. For example, in gradient-based optimization, the gradient backpropagated from the loss function applies indiscriminately to all parameters. This allows weights that were deemed redundant and set to zero during the sparsification phase to regain non-zero values, thus disrupting the established sparsity pattern and reducing the model's true sparsity. Simultaneously, the quantized weights may exceed the intended low-precision representation range during parameter updates, causing an "inverse rebound" in effective numerical accuracy.

[0019] As mentioned above, the aforementioned schemes represent different technical paths for balancing performance and efficiency in post-processing model sparsity quantization, but each may have some inherent drawbacks. To address this, this application proposes an innovative "progressive hierarchical constraint fine-tuning" scheme. This scheme adopts an overall process of "sparse first, then quantize," and introduces a collaborative mechanism between "progressive sparsity and quantization" and "hierarchical constraint fine-tuning" within this framework.

[0020] The various embodiments of this application have some or all of the following features:

[0021] 1. Layered independent fine-tuning: Fine-tuning is performed separately for each network layer of the model to adapt to the different repair needs of each layer;

[0022] 2. Lightweight data adaptation: Only a small amount of data covering the core scenarios of the task is used for fine-tuning, significantly saving resources;

[0023] 3. Structural constraint protection: Constraints are applied during fine-tuning (e.g., the positions of zero-value parameters generated by sparsity and the precision format after quantization can be fixed) to ensure that the effects of compression (e.g., sparsity and quantization) are not compromised;

[0024] 4. Progressive sparsity mechanism: Gradually increase the sparsity rate, and perform layer-by-layer fine-tuning after each sparsity step to repair the current damage, for example, following the operation of "2 times sparsity → fine-tuning → 4 times sparsity → fine-tuning → 8 times sparsity → fine-tuning";

[0025] 5. Progressive quantization mechanism: Gradually reduce precision, and perform layered constraint fine-tuning after each quantization step to repair the current corruption, for example, following the operation of "8-bit quantization → fine-tuning → 4-bit quantization → fine-tuning";

[0026] 6. Process sequence control: Follow the order of "completing all progressive sparse processes first, and then starting the progressive quantization process" to reduce mutual interference between the two types of operations and improve optimization stability.

[0027] By employing a "progressive hierarchical constraint fine-tuning" scheme, embodiments of the present invention achieve significant reduction in model storage consumption and computational complexity while effectively maintaining or even restoring the model's performance on the target task, thereby providing a feasible technical path for efficiently and reliably deploying large models in resource-constrained environments such as edge devices.

[0028] Figure 1 A flowchart of a method 100 for sparse quantization of a model according to an embodiment of this application is shown. The method 100 for sparse quantization of a model includes the following operations.

[0029] In operation S110, a fine-tuning dataset is generated based on data associated with the target task of the pre-trained model to be optimized. In some embodiments, the pre-trained model can be a large language model (such as the GPT series models based on the Transformer architecture) or a computer vision model (such as the ResNet series models based on convolutional neural networks (CNNs)). Data associated with the target task refers to data in the specific scenario where the model is to be applied; for example, for text generation tasks, it can be a relevant corpus; for image recognition tasks, it can be a set of labeled images. In some embodiments, the fine-tuning dataset is configured to contain a subset of samples covering the core scenarios of the target task. For example, in some embodiments, the amount of data in the fine-tuning dataset is only 5% to 10% of the original full training data of the model, and it is configured to cover the core scenarios of the target task to ensure the efficiency and representativeness of subsequent fine-tuning. This operation S110 provides an accurate and low-cost data foundation for subsequent incremental optimization.

[0030] In operation S120, multiple rounds of progressive sparsity processing are performed on the pre-trained model. In embodiments of this application, a high-sparsity objective (e.g., 8x sparsity, i.e., retaining 12.5% ​​of the parameters) can be decomposed into multiple gentle progressive stages to avoid irreversible model damage caused by a single drastic sparsity. Each round of the multiple rounds of progressive sparsity processing follows a cyclical pattern of "sparse operation - hierarchical fine-tuning - structural constraints".

[0031] Figure 2 An example flowchart of a multi-round progressive sparsity processing 200 (corresponding to operation S120) according to an embodiment of this application is shown. It should be noted that the technical solution of this application is not limited. Figure 2 The operation shown is "2x sparsity → fine-tuning → 4x sparsity → fine-tuning → 8x sparsity → fine-tuning". For example, progressive sparsity processing can have more or fewer rounds.

[0032] For each round of asymptotic sparsity processing, a sparsity operation is first performed (e.g., ...). Figure 2The operation S121 shows a 2x sparsity reduction. This operation can employ an unstructured sparsity algorithm to remove a portion of the less important parameters in the model (resetting their weights to zero) based on criteria such as the absolute value of the weights or their contribution. For example, for 2x sparsity reduction, the 50% of parameters with the lowest absolute weights in the model can be set to zero.

[0033] After performing a sparsity operation, the sparsed model is fine-tuned. Specifically, fine-tuning includes performing hierarchical fine-tuning and applying sparse structure constraints (as shown in operations S122 and S123). Before operation S122, for example, during or before the data preparation stage of operation S110, the pre-trained model can be pre-divided into multiple independently operable network layers based on the network structure characteristics of the pre-trained model (e.g., Transformer layers, CNN convolutional blocks). In operation S122, using the fine-tuning dataset generated in operation S110, model fine-tuning is performed independently for each network layer. This hierarchical design allows for precise parameter fixing based on the differences in sparsity sensitivity of different layers. For example, when optimizing a Transformer model such as GPT-2, it can first be divided into multiple independently operable functional layers based on its network structure, including, for example, word embedding layers, N identical Transformer blocks (each block can be further divided into multi-head attention sub-layers and feedforward network sub-layers), and finally, the language model output head.

[0034] During the fine-tuning process in operation S122, a sparse structure constraint is applied as shown in operation S123. This constraint ensures that during the fine-tuning iterations, the positions of zero-valued parameters generated by the current sparse operation are fixed (i.e., their weights remain zero and do not participate in updates), and only non-zero parameters are allowed to be adjusted. This prevents the fine-tuning process from destroying the already constructed sparse structure. For example, during the fine-tuning process, a binary mask corresponding to the current sparse structure can be applied, which can force the gradients corresponding to all zero-valued parameters to zero, thereby strictly ensuring that the weights at these positions are not updated.

[0035] After completing one round of "sparse operations - hierarchical fine-tuning - structural constraints", the model processed and fine-tuned in the previous round can be used as a basis for the next round of sparse processing at a higher scale (e.g., ...). Figure 2 The operations S124 show the 4x sparsity processing and S127 show the 8x sparsity processing. Each round of sparsity operations further removes low-contribution parameters from the previous round's structure. For example, the 4x sparsity processing, based on the 2x sparsity structure, sets the 50% of parameters with the lowest absolute weight (i.e., 25% of the total parameters) to zero; the 8x sparsity processing, based on the 4x sparsity structure, sets the 50% of parameters with the lowest absolute weight (i.e., 12.5% ​​of the total parameters) to zero. Each round is followed immediately by targeted hierarchical fine-tuning (e.g., ...). Figure 2 Operations S125 and S128 (as shown in the layered fine-tuning) and applying corresponding structural constraints (such as...) Figure 2 (The sparse structure constraints are shown in operations S126 and S129). Through this "progressive" strategy, the model gradually adapts to higher sparsity, while avoiding cumulative performance collapse through timely, hierarchical repairs.

[0036] In some embodiments, after completing multiple rounds of progressive sparsity processing S120, pre-quantization preparation can be performed. In some embodiments, pre-quantization preparation may include: confirming that the model after multiple rounds of progressive sparsity processing meets preset performance and structural stability requirements; and adjusting the fine-tuning dataset to adapt its data distribution to the characteristics of quantization processing. For example, the data distribution can be optimized for the characteristics of the quantization task by supplementing data samples that are sensitive to changes in numerical precision, so that the subsequent quantization fine-tuning process focuses more on learning and compensating for errors introduced by quantization mapping.

[0037] Next, in operation S130, multi-round progressive quantization processing begins. Similar to the sparsity processing described above, the quantization process can also employ a progressive strategy, decomposing low-precision quantization (e.g., quantizing from 32-bit floating-point to 4-bit) into multiple operations with progressively decreasing precision (e.g., first quantizing to 8 bits, then to 4 bits) to mitigate the impact of a sudden drop in precision. Each round of multi-round progressive quantization follows a cyclical pattern of "quantization operation - hierarchical fine-tuning - double constraint".

[0038] Figure 3 An example flowchart of a multi-round progressive quantization process 300 (corresponding to operation S130) according to an embodiment of this application is shown. It should be noted that the technical solution of this application is not limited to this. Figure 3 The operation shown is "8-bit quantization → fine-tuning → 4-bit quantization → fine-tuning". For example, there can be progressive quantization processing with more or fewer rounds.

[0039] For each round of incremental quantization, first, perform a quantization operation (e.g., as shown in the image). Figure 3 The operation S131 (8-bit quantization) maps the model's weight parameters from a high-precision format (such as 32-bit floating-point numbers) to a lower integer precision (such as 8-bit integers). In some embodiments, algorithms such as symmetric quantization can be used to preserve the data distribution characteristics.

[0040] After a quantization operation is completed, the quantized model is fine-tuned. Specifically, fine-tuning includes performing hierarchical fine-tuning and applying dual structural constraints (as shown in operations S132 and S133). In operation S132, using the fine-tuning dataset described above, model fine-tuning is performed independently for each network layer.

[0041] During the fine-tuning process in operation S132, a dual structural constraint is applied as shown in operation S133. The dual structural constraint includes: a sparsity structural constraint (as described above) to maintain the positions of all zero-valued parameters obtained from the sparse phase; and a quantization structural constraint to preserve the numerical precision format of the weight parameters determined by the quantization operation during fine-tuning (e.g., maintaining an 8-bit integer representation without reverting to high precision). This dual structural constraint ensures that all hardware efficiency gains from sparsity and quantization are fully preserved while guaranteeing the restoration of model accuracy.

[0042] After completing one round of "quantization operation - hierarchical fine-tuning - double constraint" cycle, the next round of lower-precision quantization can be performed based on the model processed and fine-tuned in the previous round (e.g., ...). Figure 3 The operation S134 shows the 4-bit quantization process) and the corresponding layered fine-tuning (such as... Figure 3 The operation S135 shows the layered fine-tuning) and the application of dual structural constraints (such as Figure 3 (The operation S136 shows the dual structural constraints). Through this "progressive" strategy, an optimized model that simultaneously possesses high sparsity, low accuracy, and meets performance standards can be obtained.

[0043] In some embodiments, after completing the multi-round progressive quantization process, performance verification can be performed. Specifically, performance verification may include: performing performance tests on the optimized final model based on the original target task to obtain performance metrics, such as calculating the perplexity of text generation or the accuracy of image recognition; and detecting the sparsity and quantization accuracy of the model to confirm that its sparse structure and quantization format have not been destroyed after the complete process, thereby verifying that the sparse structure constraints and quantization structure constraints are maintained. In some embodiments, the performance metrics can be compared with the corresponding metrics of at least one reference model (e.g., a model that has only undergone sparsification and quantization processing but has not undergone any fine-tuning) to objectively evaluate and confirm whether the performance recovery of the model after completing the multi-round progressive quantization process meets the preset requirements.

[0044] In some embodiments, the optimized model can be loaded onto target hardware, such as an edge computing device, mobile terminal, IoT device, or embedded system, and run on the target hardware to perform the target task.

[0045] By implementing the above method, the embodiments of the present invention achieve a significant reduction in model storage consumption and computational complexity while effectively maintaining or even restoring the model's performance on the target task, thereby providing a feasible technical path for efficiently and reliably deploying large models in resource-constrained environments such as edge devices.

[0046] Figure 4 This application illustrates an implementation method based on an embodiment of the present application. Figure 1 An exemplary system diagram of the neural network (NN) processing unit 400 of the method 100 shown. Figure 4 The neural network processing unit 400 may include a neural network accelerator or neural network processing unit (NPU), such as a dedicated microprocessor designed to accelerate machine learning and artificial intelligence (AI) tasks. Unlike traditional central processing units (CPUs) and graphics processing units (GPUs), accelerators or NPUs are specifically optimized for neural network operations such as convolution computations and vector operations. Those skilled in the art will understand that the algorithms mentioned in this disclosure are not limited to those described herein. Figure 4 The processing units shown are also applicable to other processing units, such as CPUs, GPUs, and TPUs.

[0047] Figure 4 The NN processing unit 400 shown includes multiple processing entities (PEs) designed to provide maximum parallelism to accelerate neural network operations. These PEs are organized in a 2D mesh network and connected via a router network (in... Figure 4 The interconnect is denoted as "R" in Chinese. Furthermore, the NN processing unit 400 incorporates double data rate (DDR) memory modules and caches, such as the last-level cache (LLC), to support efficient data storage and retrieval. Figure 4 The NN processing unit 400 described herein is illustrative only and may include more, fewer, or alternative components. The NN processing unit 400 may be designed as a reconfigurable device, such as a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). The method 100 introduced in this application may be implemented in various PEs.

[0048] In some embodiments, each PE has a sorting circuit (not shown in the figure) that supports sorting functionality, which performs sorting and taking the first k values. Additionally, in some embodiments, each PE also has a sparse processing unit (SPU) (not shown in the figure) that can execute sparse matrix multiplication instructions as described above.

[0049] In some embodiments, to optimize resource utilization and enhance the parallel processing capability of the NN processing unit 400, the PEs within the 2D mesh network can be divided into multiple parts, each referred to as a processor core. For example... Figure 4As shown, each group of 16 PEs (arranged in a 4×4 grid) constitutes a processor core, resulting in a total of four processor cores across the 64 PEs in the NN processing unit 400. Each processor core is equipped with dedicated DDR memory, LLC, and control circuitry, including a RISC-V Vector Unit (RVV) responsible for vector processing, a Core-Level Scheduler (CoLS) managing the execution and synchronization of multiple PEs, and an Instruction Scheduling Unit (IDU) that distributes instructions to the various execution units within the accelerator. This architecture of the NN processing unit 400 enables all four processor cores 410 (i.e., four groups of PEs) to operate concurrently, ensuring efficient parallel processing.

[0050] In some embodiments, these processor cores 410 are also organized into an on-chip network (NoC) for inter-core communication. For example, in Figure 4 In this architecture, four Processor Providers (PEs) are arranged in a ring-shaped NoC, facilitating seamless communication between cores and improving overall compute throughput. The ring-shaped NoC architecture comprises a ring arrangement of processor cores 410, where data packets travel along a unidirectional or bidirectional ring, traversing each processor core 410 until they reach their destination. The ring-shaped NoC can be used for data communication between DDR and PCIe, or for PEs to read data from DDR belonging to other cores. In addition to the ring-shaped NoC, PEs can also be arranged in a 2D mesh NoC to manage data communication between PEs.

[0051] also, Figure 4 The ring-shaped NoC architecture is scalable, allowing additional processor cores or PEs to be added to the ring-shaped NoC without significantly increasing network complexity. This flexibility supports the expansion of NN processing units to accommodate larger neural network models or additional computational tasks required.

[0052] Figure 5 An example computing device 500 in which a processor core according to an embodiment of the present disclosure can be used is shown. Figure 5 As shown, computing device 500 may include bus 502 or other communication mechanism for transmitting information, and one or more hardware processors 504 coupled to bus 502 for processing information. The one or more hardware processors 504 may include, for example, one or more general-purpose microprocessors.

[0053] like Figure 5As shown, in some embodiments, computing device 500 may further include main memory 506 coupled to bus 502. Main memory 506 is used to store information and instructions executed by one or more processors 504, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 506 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 504. When these instructions are stored in storage media accessible to one or more processors 504, they can cause computing device 500 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 508 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or their networking versions.

[0054] like Figure 5 As shown, in some embodiments, computing device 500 may further include one or more communication interfaces or network interfaces 510 coupled to bus 502. Network interface 510 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 510 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0055] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0056] Each of the processes, methods, and algorithms described in the preceding sections may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented, partially or fully, in dedicated circuit systems.

[0057] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0058] Specific embodiments further provide a system including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the system to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0059] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (collectively referred to below as the "Service System") that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0060] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this application. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0061] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0062] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0063] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0064] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0065] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0066] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this application. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this application as defined by the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0067] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0068] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes can be made to these embodiments without departing from the broad scope of embodiments of this application. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0069] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this application. Therefore, "implementation" is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their equivalents in their full scope.

Claims

1. A method for sparse quantization of a model, characterized in that, The method includes: A fine-tuning dataset is generated based on data associated with the target task of the pre-trained model, wherein the target task is a text generation task or an image recognition task, and the data is corpus data or image data. Perform multiple rounds of progressive sparsity processing on the pre-trained model, wherein each round of progressive sparsity processing includes: Perform sparse operations; The model obtained after this sparse operation is then fine-tuned hierarchically using the fine-tuning dataset. Sparse structure constraints are applied during the hierarchical fine-tuning process; After the multi-round progressive sparsity processing, the model after the multi-round progressive sparsity processing is subjected to multi-round progressive quantization processing, wherein each round of the multi-round progressive quantization processing includes: Perform quantification operations; The model obtained after this quantization operation is then fine-tuned hierarchically using the fine-tuning dataset. During the hierarchical fine-tuning process, both sparse structure constraints and quantization structure constraints are applied. The sparse structure constraints are configured to fix the position of the zero-value weight parameters generated by the sparse operation during the hierarchical fine-tuning process of the multi-round progressive sparse processing and the multi-round progressive quantization processing. The quantization structure constraints are configured to maintain the numerical precision format of the weight parameters determined by the quantization operation unchanged during the hierarchical fine-tuning process of the multi-round progressive quantization processing.

2. The method according to claim 1, characterized in that, The fine-tuned dataset is configured to contain a subset of samples covering the core scenarios of the target task.

3. The method according to claim 1, characterized in that, The multi-round progressive sparsity processing includes sequential execution of 2x sparsity processing, 4x sparsity processing, and 8x sparsity processing, wherein each round of sparsity processing is performed based on the model after the previous round of sparsity processing.

4. The method according to claim 3, characterized in that, The 2x sparsity processing includes setting the 50% of parameters with the lowest absolute weights in the model to zero; the 4x sparsity processing includes setting the 50% of parameters with the lowest absolute weights in the remaining parameters to zero, based on the 2x sparsity processing; the 8x sparsity processing includes setting the 50% of parameters with the lowest absolute weights in the remaining parameters to zero, based on the 4x sparsity processing.

5. The method according to claim 1, characterized in that, The multi-round progressive quantization process includes 8-bit quantization and 4-bit quantization processes executed sequentially, wherein each round of quantization is performed based on the model of the previous round of quantization.

6. The method according to claim 5, characterized in that, The 8-bit quantization process includes mapping the model's weight parameters from a 32-bit floating-point representation to an 8-bit integer representation; the 4-bit quantization process includes further mapping the weight parameters represented by the 8-bit integers to a 4-bit integer representation.

7. The method according to any one of claims 1-6, characterized in that, The method further includes: dividing the pre-trained model into multiple network layers according to the network structure characteristics of the pre-trained model, wherein the layer-by-layer fine-tuning is performed independently for each of the multiple network layers.

8. The method according to any one of claims 1-6, characterized in that, Before performing the multi-round progressive quantization process, the method further includes: The model after the multiple rounds of progressive sparsity processing was confirmed to meet the preset performance and structural stability requirements; and The fine-tuned dataset is adjusted to adapt its data distribution to the characteristics of quantization processing.

9. The method according to any one of claims 1-6, characterized in that, After completing the multi-round progressive quantization process, the method further includes: Based on the stated objective task, the model after completing the multiple rounds of progressive quantization processing is tested to obtain performance metrics; and The sparse structure and quantization accuracy of the model after the multi-round progressive quantization process are detected to verify that the sparse structure constraints and the quantization structure constraints are maintained.

10. The method according to claim 9, characterized in that, The method further includes: The performance metrics are compared with the corresponding metrics of at least one reference model to confirm that the performance of the model after the multi-round progressive quantization process meets the preset requirements.

11. The method according to any one of claims 1-6, characterized in that, The amount of data in the fine-tuned dataset accounts for 5% to 10% of the original training data of the pre-trained model.

12. The method according to any one of claims 1-6, characterized in that, The method further includes loading the model after the multi-round progressive quantization process is completed onto the target hardware.

13. The method according to claim 12, characterized in that, The target hardware includes edge computing devices, mobile terminals, Internet of Things devices, or embedded systems.

14. A system for sparse quantization of a model, characterized in that, The system includes: processor; The memory stores instructions that, when executed by the processor, perform the following operations: A fine-tuning dataset is generated based on data associated with the target task of the pre-trained model, wherein the target task is a text generation task or an image recognition task, and the data is corpus data or image data. Perform multiple rounds of progressive sparsity processing on the pre-trained model, wherein each round of progressive sparsity processing includes: Perform sparse operations; The model obtained after this sparse operation is then fine-tuned hierarchically using the fine-tuning dataset. Sparse structure constraints are applied during the hierarchical fine-tuning process; After completing the multi-round progressive sparsity processing, the model after multi-round progressive sparsity processing is subjected to multi-round progressive quantization processing, wherein each round of the multi-round progressive quantization processing includes: Perform quantification operations; The model obtained after this quantization operation is then fine-tuned hierarchically using the fine-tuning dataset. During the hierarchical fine-tuning process, both sparse structure constraints and quantization structure constraints are applied. The sparse structure constraints are configured to fix the position of the zero-value weight parameters generated by the sparse operation during the hierarchical fine-tuning process of the multi-round progressive sparse processing and the multi-round progressive quantization processing. The quantization structure constraints are configured to maintain the numerical precision format of the weight parameters determined by the quantization operation unchanged during the hierarchical fine-tuning process of the multi-round progressive quantization processing.

15. A computer program product, characterized in that, Includes instructions that, when executed by a processor, perform the method according to any one of claims 1-13.

16. A non-transitory computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, perform the method according to any one of claims 1-13.

Images