Method, device and computer storage medium for neural network model compression

Abstract

A method of neural network model compression performed by at least one processor, the method comprising receiving an input neural network and an input mask, and reducing parameters of the input neural network using a deep neural network. The deep neural network is trained by selecting, from a plurality of blocks of input weights of the deep neural network that are masked by the input mask, a pruning microstructure block to be pruned; pruning the input weights based on the selected pruning microstructure block; selecting, from the plurality of blocks of input weights that are masked by the input mask, a unification microstructure block to be unified; and unifying, based on the selected unification microstructure block, a plurality of weights in one or more of the plurality of blocks of pruned input weights to obtain pruned and unified input weights of the deep neural network.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 17 / 319,313, filed May 13, 2021, with the United States Patent and Trademark Office, which claims priority to U.S. Provisional Patent Application No. 63 / 040,216, filed June 17, 2020; U.S. Provisional Patent Application No. 63 / 040,238, filed June 17, 2020; and U.S. Provisional Patent Application No. 63 / 043,082, filed June 23, 2020, the entire contents of which are incorporated herein by reference. Background Technology

[0003] The success of Deep Neural Networks (DNNs) in a wide range of video applications, such as semantic classification, object detection / recognition, object tracking, and video quality enhancement, has created a demand for compressed DNN models. Therefore, the Motion Picture Experts Group (MPEG) is actively working on the encoding representation of the Neural Network standard (NNR), which is used to encode DNN models to save on storage and computational costs. Summary of the Invention

[0004] According to an embodiment, a method for compressing a neural network model, executed by at least one processor, includes receiving an input neural network and an input mask; and reducing the parameters of the input neural network using a deep neural network trained by: selecting pruning micro-structure blocks to be pruned from a plurality of blocks of input weights masked by the input mask; pruning the input weights based on the selected pruning micro-structure blocks; selecting unifying micro-structure blocks to be unified from a plurality of blocks of input weights masked by the input mask; and unifying a plurality of weights in one or more blocks of pruned input weights based on the selected unifying micro-structure blocks to obtain pruned and unified input weights of the deep neural network. The method further includes: obtaining an output neural network with reduced parameters based on the input neural network and the pruned and unified input weights of the deep neural network.

[0005] According to an embodiment, an apparatus for compressing a neural network model includes: at least one memory configured to store program code, and at least one processor configured to read the program code and operate according to the instructions of the program code. The program code includes receiving code configured to cause the at least one processor to receive an input neural network and an input mask; and reduction code configured to cause the at least one processor to reduce the parameters of the input neural network using a deep neural network; the deep neural network is trained by: selecting a pruned microstructure block to be pruned from a plurality of blocks of input weights masked by the input mask; pruning the input weights based on the selected pruned microstructure block; selecting a unified microstructure block to be unified from a plurality of blocks of input weights masked by the input mask; and unifying a plurality of weights in one or more blocks of the pruned input weights based on the selected unified microstructure block to obtain pruned and unified input weights of the deep neural network. The program code further includes: acquisition code, which is configured to cause the at least one processor to output an output neural network with reduced parameters based on the pruned and unified input weights of the input neural network and the deep neural network.

[0006] According to an embodiment, a non-transitory computer-readable medium stores instructions that, when executed by at least one processor for neural network model compression, cause the at least one processor to: receive an input neural network and an input mask; reduce the parameters of the input neural network using a deep neural network trained by: selecting pruned microstructure blocks to be pruned from a plurality of blocks of input weights masked by the input mask; pruning the input weights based on the selected pruned microstructure blocks; selecting unified microstructure blocks to be unified from a plurality of blocks of input weights masked by the input mask; and unifying a plurality of weights in one or more blocks of the pruned input weights based on the selected unified microstructure blocks to obtain pruned and unified input weights of the deep neural network. The instructions, when executed by the at least one processor, further cause the at least one processor to obtain an output neural network with reduced parameters based on the input neural network and the pruned and unified input weights of the deep neural network. Attached Figure Description

[0007] Figure 1 This is a diagram illustrating the environment in which the methods, apparatus, and systems described herein can be implemented according to embodiments.

[0008] Figure 2 yes Figure 1A block diagram of example components of one or more devices.

[0009] Figure 3 This is a functional block diagram of a system for neural network model compression according to an embodiment.

[0010] Figure 4A This is a functional block diagram of a training apparatus for compressing a neural network model using micro-structured weight pruning, according to an embodiment.

[0011] Figure 4B This is a functional block diagram of a training apparatus for compressing a neural network model using microstructured weight pruning, according to other embodiments.

[0012] Figure 4C This is a functional block diagram of a training device for compressing a neural network model with uniform weights, according to some other embodiments.

[0013] Figure 4D This is a functional block diagram of a training apparatus for compressing a neural network model using microstructured weight pruning and weight unification, according to some other embodiments.

[0014] Figure 4E This is a functional block diagram of a training apparatus for compressing a neural network model using microstructured weight pruning and weight unification, according to some other embodiments.

[0015] Figure 5 This is a flowchart illustrating a method for compressing a neural network model using microstructured weight pruning and weight unification, as described in the embodiments.

[0016] Figure 6 This is a block diagram of an apparatus for compressing a neural network model using microstructured weight pruning and weight unification, according to an embodiment. Detailed Implementation

[0017] This disclosure relates to neural network model compression. More specifically, the methods and apparatus described herein relate to neural network model compression employing microstructured weight pruning and weight unification.

[0018] The embodiments described herein include a method and apparatus for compressing DNN models by using microstructured weight pruning regularization within an iterative network retraining / finetuning framework. The pruning loss is jointly optimized to connect to the original network training objective through an iterative retraining / finetuning process.

[0019] The embodiments described herein further include a method and apparatus for compressing a DNN model by using structured unified regularization within an iterative network retraining / fine-tuning framework. The weight unification loss includes compression ratio loss, unified distortion loss, and computational speed loss. The weight unification loss is jointly optimized with the original network training objective through an iterative retraining / fine-tuning process.

[0020] The embodiments described herein further include a method and apparatus for compressing a DNN model by using microstructured joint weight pruning and weight unification regularization within an iterative network retraining / fine-tuning framework. The pruning loss, the unification loss, and the original network training objective are jointly optimized through the iterative retraining / fine-tuning process.

[0021] Several approaches exist for learning compressed DNN models. The goal is to remove unimportant weight coefficients, assuming that smaller weight values ​​indicate lower importance and that removing these weights has less impact on prediction performance. To achieve this, several network pruning methods have been proposed. For example, unstructured weight pruning adds a sparsity-promoting regularization term to the network training objective, resulting in unstructured zero-value weights. This reduces model size but does not reduce inference time. Structured weight pruning randomly forces pruning of the entire weight structure, such as rows or columns. Removed rows or columns do not participate in inference computation, reducing model size and inference time. However, removing the entire weight structure (such as rows and columns) can lead to a significant performance degradation of the original DNN model.

[0022] Several network pruning methods incorporate sparsity-enhancing regularization terms into the network training objective. Unstructured weight pruning methods add sparsity-enhancing regularization terms to the network training objective, resulting in unstructured, zero-valued weights. Structured weight pruning methods randomly force pruning of the selected weight structure, such as rows or columns. From the perspective of compressed DNN models, after learning a compressed network model, the weight coefficients can be further compressed through quantization and subsequent entropy encoding. This further compression process can significantly reduce the storage size of the DNN model, enabling its deployment on mobile devices, chips, etc.

[0023] The embodiments described herein include a method and apparatus for microstructured weight pruning, aimed at reducing model size and accelerating inference computation with minimal loss of the original DNN model's predictive performance. An iterative network retraining / refining framework is employed to jointly optimize the original training objective and the weight pruning loss. Weight coefficients are pruned according to small microstructures aligned with the underlying hardware design, thereby significantly reducing model size, preserving the original objective's predictive performance, and greatly accelerating inference computation. This method and apparatus can be used to compress pre-trained dense DNN models. They can also serve as additional processing modules to further compress pre-pruned sparse DNN models using other unstructured or structured pruning methods.

[0024] The embodiments described herein further include: a method and apparatus for structured weight unification regularization, aimed at improving compression efficiency in subsequent compression processes. Employing an iterative network retraining / refining framework, the original training objective and a weight unification loss, including compression ratio loss, unification distortion loss, and computation speed loss, are jointly optimized. This ensures that the learned network weight coefficients maintain the original target performance, are suitable for further compression, and accelerate computation using the learned weight coefficients. This method and apparatus can be used to compress pre-trained DNN models. It can also serve as an additional processing module for further compression of arbitrary pruned DNN models.

[0025] The embodiments described herein include a method and apparatus for jointly performing microstructured weight pruning and weight unification, aimed at improving compression efficiency and accelerating inference computation in subsequent compression processes. An iterative network retraining / refining framework is employed to jointly optimize the original training objective, weight pruning loss, and weight unification loss. Weight coefficients are pruned or unified based on small microstructures; the learned weight coefficients maintain the original objective performance, are suitable for further compression, and accelerate computation using the learned weight coefficients. This method and apparatus can be used to compress pre-trained dense DNN models. This method and apparatus can also serve as an additional processing module to further compress pre-pruned sparse DNN models using other unstructured or structured pruning methods.

[0026] Figure 1 This is a schematic diagram of an environment 100 in which the methods, apparatus and systems described herein can be implemented, according to an embodiment.

[0027] like Figure 1As shown, environment 100 may include user equipment 110, platform 120, and network 130. The devices in environment 100 can be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0028] User equipment 110 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information related to platform 120. For example, user equipment 110 may include computing devices (e.g., desktop computers, laptop computers, tablet computers, handheld computers, smart speakers, servers, etc.), mobile phones (e.g., smartphones, cordless phones, etc.), wearable devices (e.g., smart glasses or smartwatches), or similar devices. In some embodiments, user equipment 110 may receive information from and / or send information to platform 120.

[0029] Platform 120 includes one or more devices as described elsewhere herein. In some embodiments, platform 120 may include a cloud server or a group of cloud servers. In some embodiments, platform 120 may be designed to be modular, allowing software components to be swapped in or out. This allows platform 120 to be easily and / or quickly reconfigured for different uses.

[0030] In some implementations, as shown in the figures, platform 120 may be hosted in a cloud computing environment 122. It is worth noting that while the implementations described herein describe platform 120 as hosted in a cloud computing environment 122, in some implementations, platform 120 may not be cloud-based (i.e., may be implemented outside of a cloud computing environment) or may be partially cloud-based.

[0031] The cloud computing environment 122 includes the environment of the hosting platform 120. The cloud computing environment 122 can provide services such as computing, software, data access, and storage, without requiring end users (e.g., user equipment 110) to know the physical location and configuration of the systems and / or devices of the hosting platform 120. As shown in the figure, the cloud computing environment 122 may include a set of computing resources 124 (collectively referred to as "computing resources 124" and individually as "computing resource 124").

[0032] Computing resource 124 includes one or more personal computers, workstations, server devices, or other types of computing and / or communication devices. In some embodiments, computing resource 124 may host platform 120. Cloud resources may include computing instances executing in computing resource 124, storage devices provided in computing resource 124, data transmission devices provided by computing resource 124, etc. In some embodiments, computing resource 124 may communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0033] Further as Figure 1 As shown, computing resources 124 include a set of cloud resources, such as one or more applications (“APP”) 124-1, one or more virtual machines (“VM”) 124-2, virtualized storage (“VS”) 124-3, one or more hypervisors (“HYP”) 124-4, etc.

[0034] Application 124-1 includes one or more software applications that can be provided to, or accessed by, user device 110 and / or platform 120. Application 124-1 does not require the installation and execution of any software applications on user device 110. For example, application 124-1 may include software associated with platform 120, and / or any other software available through cloud computing environment 122. In some implementations, an application 124-1 may send / receive information to or from one or more other applications 124-1 via virtual machine 124-2.

[0035] Virtual machine 124-2 includes a software implementation of a machine (e.g., a computer) that executes programs, similar to a physical machine. Virtual machine 124-2 can be a system virtual machine or a process virtual machine, depending on the extent to which virtual machine 124-2 uses and corresponds to any real machine. A system virtual machine can provide a complete system platform that supports the execution of a complete operating system (“OS”). A process virtual machine can execute a single program and can support a single process. In some implementations, virtual machine 124-2 can execute on behalf of a user (e.g., user device 110) and can manage the infrastructure of cloud computing environment 122, such as data management, synchronization, or long-term data transfer.

[0036] Virtualized storage 124-3 includes one or more storage systems and / or one or more devices that utilize virtualization technology within the storage systems or devices of computing resource 124. In some implementations, the type of virtualization within the context of the storage system may include block virtualization and file virtualization. Block virtualization may refer to the abstraction (or separation) of logical storage from physical storage so that the storage system can be accessed without regard to physical storage or heterogeneous architecture. Separation allows storage system administrators to flexibly manage end-user storage. File virtualization can eliminate the dependency between data accessed at the file level and the location of physical storage files. This can optimize storage usage, server consolidation, and / or performance for non-disruptive file migration.

[0037] Hypervisor 124-4 provides hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to run simultaneously on a host computer such as computing resource 124. Hypervisor 124-4 can provide a virtual operating platform to the guest operating systems and manage their execution. Multiple instances of various operating systems can share virtualized hardware resources.

[0038] Network 130 includes one or more wired and / or wireless networks. For example, network 130 may include cellular networks (e.g., fifth-generation (5G) networks, Long-Term Evolution (LTE) networks, third-generation (3G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTNs)), private networks, self-organizing networks, intranets, the Internet, fiber-optic networks, etc., and / or combinations of these or other types of networks.

[0039] Figure 1 The number and arrangement of devices and networks shown are provided as an example. In reality, with... Figure 1 Compared to the devices and / or networks shown, there can be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently. Furthermore, Figure 1The two or more devices shown can be implemented within a single device, or Figure 1 The single device shown can be implemented as multiple distributed devices. Alternatively, a group of devices in environment 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in environment 100.

[0040] Figure 2 yes Figure 1 A block diagram of example components of one or more devices. Device 200 may correspond to user device 110 and / or platform 120.

[0041] like Figure 2 As shown, device 200 may include bus 210, processor 220, memory 230, storage component 240, input component 250, output component 260 and communication interface 270.

[0042] Bus 210 includes components that allow communication between components of device 200. Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. Processor 220 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing component. In some embodiments, processor 220 includes one or more processors that can be programmed to perform functions. Memory 230 includes random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic storage, and / or optical storage) that stores information and / or instructions for use by processor 220.

[0043] Storage component 240 stores information and / or software related to the operation and use of device 200. For example, storage component 240 may include hard disks (e.g., magnetic disks, optical disks, magneto-optical disks, and / or solid-state disks), optical disks (CDs), digital versatile disks (DVDs), floppy disks, cassette tapes, magnetic tapes, and / or other types of non-volatile computer-readable media, and corresponding drives.

[0044] Input component 250 includes components that allow device 200 to receive information, such as a touchscreen display, keyboard, keypad, mouse, buttons, switches, and / or microphone. Alternatively, input component 250 may include sensors for sensing information (e.g., a Global Positioning System (GPS) component, accelerometer, gyroscope, and / or actuator). Output component 260 includes components that provide output information from device 200, such as a display, speaker, and / or one or more light-emitting diodes (LEDs).

[0045] Communication interface 270 includes transceiver-like components (e.g., a transceiver and / or separate receiver and transmitter) that enable device 200 to communicate with other devices, for example, via a wired connection, a wireless connection, or a combination of wired and wireless connections. Communication interface 270 may allow device 200 to receive information from and / or provide information to another device. For example, communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0046] Device 200 can perform one or more processes described herein. Device 200 can perform these processes in response to processor 220 executing software instructions stored in a non-volatile computer-readable medium (e.g., memory 230 and / or storage component 240). Computer-readable medium is defined herein as a non-volatile memory device. A memory device includes storage space within a single physical storage device or storage space distributed across multiple physical storage devices.

[0047] Software instructions can be read into memory 230 and / or storage component 240 from another computer-readable medium or from another device via communication interface 270. When executed, the software instructions stored in memory 230 and / or storage component 240 can cause processor 220 to perform one or more processes described herein. Alternatively or additionally, hardware wiring circuitry may be used in place of or in combination with the software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software.

[0048] Figure 2 The number and arrangement of components shown are provided as an example. In practice, compared to Figure 2 The components shown in the diagram indicate that the device 200 may include additional components, fewer components, different components, or components arranged differently. Alternatively or additionally, a set of components of the device 200 (e.g., one or more components) may perform one or more functions described as being performed by another set of components of the device 200.

[0049] The method and apparatus for compressing neural network models using microstructured weight pruning and weight unification will now be described in detail.

[0050] Figure 3 This is a functional block diagram of a system 300 for neural network model compression according to an embodiment.

[0051] like Figure 3As shown, system 300 includes a parameter reduction module 310, a parameter approximation module 320, a reconstruction module 330, an encoder 340, and a decoder 350.

[0052] The parameter reduction module 310 reduces a set of parameters of the input neural network to obtain the output neural network. The neural network may include parameters and architecture specified by the deep learning framework.

[0053] For example, parameter reduction module 310 can sparsify (set weights to zero) and / or prune away connections in the neural network. In another example, parameter reduction module 310 can perform matrix factorization on the parameter tensor of the neural network to obtain a smaller set of parameter tensors. Parameter reduction module 310 can perform these methods sequentially; for example, it can first sparsify the weights and then factorize the resulting matrix.

[0054] The parameter approximation module 320 applies a parameter approximation technique to the parameter tensor extracted from the output neural network, which is obtained from the parameter reduction module 310. For example, this technique may include any one or any combination of quantization, transformation, and prediction. The parameter approximation module 320 outputs a first parameter tensor that has not been modified by the parameter approximation module 320, a second parameter tensor that has been modified or approximated by the parameter approximation module 320, and corresponding metadata, which is used to reconstruct the original parameter tensor that has not been modified by the parameter approximation module 320 from the modified second parameter tensor.

[0055] The reconstruction module 330 uses the corresponding metadata obtained from the parameter approximation module 320 and / or the decoder 350 to reconstruct the original parameter tensor from the modified second parameter tensor, which is obtained from the parameter approximation module 320 and / or the decoder 350. The reconstruction module 330 can then reconstruct the output neural network using the reconstructed original parameter tensor and the first parameter tensor.

[0056] Encoder 340 can perform entropy encoding on the first parameter tensor, the second parameter tensor, and the corresponding metadata obtained from parameter approximation module 320. This information can be encoded into a bitstream that reaches decoder 350.

[0057] Decoder 350 can decode the bitstream obtained from encoder 340 to obtain the first parameter tensor, the second parameter tensor, and the corresponding metadata.

[0058] System 300 can be implemented in platform 120, and Figure 3One or more modules can be executed by a device or a group of devices (such as user equipment 110) that are separate from or include the platform 120.

[0059] The parameter reduction module 310 or the parameter approximation module 320 may include a DNN, which is trained by the following training device.

[0060] Figure 4A This is a functional block diagram of a training device 400A for compressing a neural network model using microstructured weight pruning, according to an embodiment. Figure 4B This is a functional block diagram of a training device 400B for neural network model compression using microstructured weight pruning, according to other embodiments.

[0061] like Figure 4A As shown, the training device 400A includes a microstructure selection module 405, a weight pruning module 410, a network forward calculation module 415, a target loss calculation module 420, a gradient calculation module 425, and a weight update module 430.

[0062] like Figure 4B As shown, the training device 400B includes a microstructure selection module 405, a weight pruning module 410, a network forward computation module 415, a target loss computation module 420, a gradient computation module 425, and a weight update module 430. The training device 400B further includes a mask computation module 435.

[0063] set up Let Θ = {w} represent a dataset in which the target y is assigned to the input x. Let Θ = {w} denote a set of weight coefficients of the DNN (e.g., parameter reduction module 310 or parameter approximation module 320). The goal of network training is to learn the optimal set Θ of weight coefficients such that the target loss... It can be minimized. For example, in previous network pruning methods, the target loss... There are two parts: empirical data loss. Sparsity promotes regularization loss

[0064]

[0065] Where, λ R ≥0 is a hyperparameter that balances the contribution of λ to data loss and the contribution of regularization loss. R When = 0, there is only the target loss. Only empirical data loss is considered, and the pre-trained weight coefficients are dense.

[0066] The pre-trained weight coefficients Θ can be further trained through another network training process, during which the optimal set of weight coefficients can be learned, achieving further model compression and inference acceleration. Examples include microstructured pruning methods to achieve this goal.

[0067] Specifically, the microstructured weight pruning loss is defined. The microstructured weight pruning loss is optimized together with the original objective loss:

[0068]

[0069] Where, λ S ≥0 is a hyperparameter that balances the contribution of the original training objective and the contribution of the weight pruning objective. This is achieved by applying equation (2) to... Optimization is performed to obtain the optimal set of weight coefficients, which can greatly contribute to the effectiveness of further compression. Furthermore, the microstructured weight pruning loss takes into account how the convolution operation is performed as a GEMM matrix multiplication process, resulting in optimized weight coefficients that can significantly accelerate computation. It is worth noting that the weight pruning loss can be viewed as... (when λ...) R >0) or not have (when λ) R When the regularization is 0, an additional regularization term is added to the target loss. Furthermore, this method can be flexibly applied to any regularized loss. R (Θ).

[0070] To improve learning effectiveness and efficiency, an iterative optimization algorithm is executed. In the first step, the partial weight coefficients that satisfy the desired microstructure are fixed. Then, in the second step, the unfixed parts of the weight coefficients are updated by training the loss through backpropagation. By iteratively performing these two steps, more and more weights can be gradually fixed, and the joint loss can be effectively optimized incrementally.

[0071] Furthermore, in the embodiments, each layer is compressed individually, therefore It can be further written as:

[0072]

[0073] Among them, L s (W j ) is the pruning loss defined at the j-th layer, N is the total number of layers involved in the training process, and W j Let L represent the weight coefficient of the j-th layer. Again, since L is calculated independently for each layer... S (W jTherefore, the superscript j can be omitted without losing generality.

[0074] For each network layer, its weight coefficients W have a size (c i ,k1,k2,k3,c o The layer is a 5-dimensional (5D) tensor of size (h). i ,w i ,d i ,c i The layer has a 4-dimensional (4D) tensor A, and the output of this layer is a tensor of size (h). o ,w o ,d o ,c o The four-dimensional tensor B has size c. i k1, k2, k3, c o h i w i d i h o w o d o It is an integer greater than or equal to 1. When the size c i k1, k2, k3, c o h i w i d i h o w o d o When any one of the terms in M ​​is 1, the corresponding tensor is reduced to a lower dimension. Each term in each tensor is a floating-point number. Let M represent a 5D binary mask of the same size as W, where each term in M ​​is a binary number 0 / 1, indicating whether the corresponding weight coefficients are pruned / remain unchanged during pre-pruning. M is introduced in association with W to address the case where W comes from a pruned DNN model using a previous structured or unstructured pruning method, in which some connections between neurons in the network are removed from computation. When W comes from the original undpruned dense model, all terms in M ​​are 1. Output B is computed via a convolution operation ⊙ based on A, M, and W.

[0075]

[0076] l = 1, ..., h i m=1,…,w i n=1,…,d i ,l′=1,…,h o ,

[0077] m′=1,…,wo ,n′=1,…,d o v = 1, ..., c o (4)

[0078] parameter h i w i and d i (h0、w o and d o ) represents the height, weights, and depth of the input tensor A (output tensor B). Parameter c i (c o ) represents the number of input (output) channels. Parameters k1, k2, and k3 correspond to the sizes of the convolution kernels along the height, weight, and depth axes, respectively. That is, for each output channel v = 1, ..., c o The operation described in equation (4) can be viewed as a convolution with input A of size (c i The 4D weighted tensor W of k1, k2, k3) v .

[0079] The order of the summation operations in equation (4) can be changed to produce different configurations of the shapes of the input A, weights W (and mask M) to obtain the same output B. In the embodiment, two configurations are used. (1) The 5D weight tensor is reshaped to a size of (c′ i ,c′ o A 3D tensor of (k), where c′ i ×c′ o ×k=c i ×c o ×k1×k2×k3. For example, this configuration is c′. i =c i ,c′ o =c o ,k=k1×k2×k3. (2) Reshape the 5D weight tensor to a size of (c′ i ,c′ o A 2D matrix, where c′ i ×c′ o =c i ×c o ×k1×k2×k3. For example, in some embodiments, it is c′. i =c i ,c′ o =c o ×k1×k2×k3 or c′ o =c o ,c′ i =c i ×k1×k2×k3.

[0080] The desired microstructure of the weight coefficients aligns with the underlying GEMM matrix multiplication process used to implement convolution operations, thereby accelerating inference computations using the learned weight coefficients. In the embodiments, block-wise microstructures for the weight coefficients are used in each layer of the 3D reshaped weight tensor or 2D reshaped weight matrix. Specifically, in the case of the reshaped 3D weight tensor, it is divided into layers of size (g... i ,g o ,g k ) blocks, and for the case of the reshaped 2D weight matrix, it is divided into blocks of size (g i ,g o The pruning operation occurs within a 2D or 3D block, where the pruned weights in the block are set to all zero. A pruning loss for the block can be calculated to measure the error introduced by such pruning. Given this microstructure, during iteration, a portion of the weight coefficients to be pruned are determined based on the pruning loss. Then, in a second step, the pruned weights are fixed, the normal neural network training process is performed, and the remaining unfixed weight coefficients are updated via backpropagation.

[0081] Figure 4A and 4B An embodiment of the iterative retraining / fine-tuning process is illustrated, where both embodiments iteratively alternate between two steps to progressively optimize the joint loss of equation (2). Given a pre-trained DNN model with weight coefficients {W} and a mask {M} (which can be a pruned sparse model or an unpruned non-sparse model), in the first step, the microstructure selection module 405 first reshapes the weight coefficients W (and the corresponding mask M) of each layer into the desired 3D tensor or 2D matrix. Then, for each layer, the microstructure selection module 405 determines a set of pruned microstructures {b} whose weights will be pruned through a pruning micro-structure selection process. s} or pruning micro-structure block (PMB). Multiple methods exist to determine the pruning microstructure {b}. s In an embodiment, for each layer with weighting coefficients W and a mask M, for each block b in W, the pruning loss L is calculated. s (b) (e.g., the sum of the absolute values ​​of the weights in b). Given a pruning ratio p, the blocks of this layer are determined according to L. s (b) Sort in ascending order and select the top p% of blocks as the {b} to be pruned. sIn other embodiments, for each layer having a weighting coefficient W and a mask M, the pruning loss L for each block b is calculated in the same manner as described above. s (b). Given a pruning ratio p, prune all blocks in all layers according to L. s (b) Sort in ascending order and select the top p% of blocks as the {b} to be pruned. s}

[0082] After obtaining this set of pruned microstructures, the objective becomes to find a set of updated optimal weight coefficients W* and corresponding weight masks M* by iteratively minimizing the joint loss described in equation (2). Figure 4A In the first embodiment shown, for the t-th iteration, there exists a current weight coefficient W(t-1). Furthermore, a micro-structurally pruning mask P(t-1) is maintained throughout the training process. P(t-1) has the same shape as W(t-1), thereby recording whether the corresponding weight coefficient has been pruned. Then, the weight pruning module 410 calculates the pruned weight coefficients W through a weight pruning process. P (t-1), where the selected pruned microstructure of the P(t-1) mask is pruned, thereby generating the updated weighted mask M. P (t-1).

[0083] Then, in the second step, the weight update module 430 fixes the weight coefficients marked by P(t-1) as microstructured and pruned, and then adjusts W through the neural network training process. P The remaining unfixed weight coefficients in (t-1) are updated to produce updated W(t) and M(t). In one embodiment, during the network training process, the pre-pruned weight coefficients masked by the pre-trained pruning mask M are forcibly fixed (i.e., kept at zero). In another embodiment, there is no such restriction on the pre-pruned weights, and the pre-pruned weights can be reset to some value other than zero during the training process, resulting in a less sparse model associated with better prediction performance, possibly even better than the original pre-trained model.

[0084] Specifically, set Denotes the training dataset, where Can be compared with the original dataset Same, based on Obtain the pre-trained weight coefficients W. It can also be with Different datasets, but with similarities to the original dataset. The same data distribution. In the second step, the network forward computation module 415 uses the current weight coefficients W. P (t-1) and mask M P (t-1) allows each input x to pass through the current network via the Network Forward Computation process, thereby generating an estimated output. Based on the truth label y and the estimated output The target loss calculation module 420 calculates the target training loss in equation (2) through the Compute Target Loss process. Then, the gradient calculation module 425 calculates the gradient G(W) of the target loss. P (t-1)). The automatic gradient calculation methods used by deep learning frameworks such as TensorFlow or PyTorch can be used to calculate G(W). P (t-1)). Based on gradient G(W) P (t-1)) and the microstructured pruning mask P(t-1), the weight update module 430 can use back propagation and weight update process to update W through back propagation. P The unfixed weight coefficients of (t-1) are updated. The retraining process itself is also an iterative process. Multiple iterations are used to update W. P The non-fixed portion of (t-1) is updated, for example, until the target loss converges. Then, the system proceeds to the next iteration t, where a new set of pruned microstructures (and a new microstructured pruning mask P(t)) is determined through a pruned microstructure selection process, given a new pruning ratio p(t).

[0085] exist Figure 4B In the second embodiment of the training process shown, a set of updated optimal weight coefficients W* and corresponding weight masks M* are found through another iterative process. For the t-th iteration, there exists a current weight coefficient W(t-1) and mask M(t-1). Furthermore, the mask calculation module 435 calculates a microstructured pruned mask P(t-1) through a pruning mask computation process. P(t-1) has the same shape as W(t-1), and P(t-1) records whether the corresponding weight coefficient has been pruned. Then, the weight pruning module 410 calculates the pruned weight coefficients W* through a weight pruning process. P (t-1), where the selected pruned microstructure to be masked is pruned by P(t-1), thereby generating an updated weighted mask M. P(t-1).

[0086] Then, in the second step, the weight update module 430 fixes the weight coefficients marked by P(t-1) as microstructured and pruned, and then updates the remaining unfixed weight coefficients of W(t-1) through the neural network training process, thereby generating the updated W(t). Similar to... Figure 4A The first embodiment, given a training dataset In this case, the network forward computation module 415 uses the current weight coefficients W(t-1) and mask M(t-1) to pass each input x through the current network via the network forward computation process, thereby generating an estimated output. Based on the truth label y and the estimated output The target loss calculation module 420 calculates the joint training loss through the Compute Joint Loss process. The joint training loss includes the target training loss in equation (2). and residual loss res (W(t-1)):

[0087]

[0088] £ res (W(t-1)) measures the current weight W(t-1) and the target pruned weight W. P The difference between (t-1). For example, the L1 norm can be used:

[0089] £ res (W(t-1))=||W(t-1))-W P (t-1)|| (6)

[0090] Then, the gradient calculation module 425 calculates the gradient of the joint loss G(W(t-1)). Automatic gradient calculation methods used by deep learning frameworks such as TensorFlow or PyTorch can be used to calculate G(W(t-1)). Based on the gradient G(W(t-1)) and the microstructured pruning mask P(t-1), the weight update module 430 updates the unfixed weight coefficients of W(t-1) via backpropagation using a backpropagation and weight update process. The retraining process itself is also an iterative process. Multiple iterations are used to update the unfixed portion of W(t-1), for example, until the target loss converges. Then, the system enters the next iteration t, where, given the pruning ratio p(t), a new set of pruned microstructures (and a new microstructured pruning mask P(t)) is determined through a pruning microstructure selection process. Similar to the previous... Figure 4AIn one embodiment, during the training process, the weight coefficients of the pre-trained pre-pruned mask M can be forced to remain at zero, or can be set again to have non-zero values.

[0091] During the entire iteration process, in the Tth iteration, the pruned weight coefficients W can be calculated through the weight pruning process. P (T), where the selected pruned microstructure to be masked is pruned by P(T) to produce an updated weighted mask M. P (T). The W P (T) and M P (T) can be used to generate the final updated models W* and M*. For example, W* = W P (T) and M*=M·M P (T).

[0092] In the embodiment, the hyperparameter p(t) can increase its value as t increases during iteration, so that more and more weight coefficients will be pruned and fixed throughout the iterative learning process.

[0093] The goal of microstructured pruning methods is to reduce model size, accelerate computation by using optimized weights, and maintain the predictive performance of the original DNN model. It can be applied to pre-trained dense models or pre-trained sparse models pruned by previous structured or unstructured pruning methods to achieve additional compression effects.

[0094] Through an iterative retraining process, this method effectively maintains the performance of the original prediction target while pursuing compression and computational efficiency. The iterative retraining process also provides the flexibility to introduce different losses at different times, allowing the system to focus on different objectives during the optimization process.

[0095] This method can be applied to datasets with different data formats. The input / output data are 4D tensors, which can be real video clips, images, or extracted feature maps.

[0096] Figure 4C This is a functional block diagram of a training device 400C for compressing a neural network model with uniform weights, according to some other embodiments.

[0097] like Figure 4C As shown, the training device 400C includes a reshaping module 440, a weight unification module 445, a network forward calculation module 415, a target loss calculation module 420, a gradient calculation module 425, and a weight update module 450.

[0098] Sparsity facilitates regularization loss to regularize all weight coefficients, resulting in sparse weights with a weak relationship to inference efficiency or computation speed. From another perspective, after pruning, the sparse weights can be further trained through another network process, during which the optimal set of weight coefficients can be learned, thereby improving the efficiency of further model compression.

[0099] Weighted uniform loss It is optimized along with the original target loss:

[0100]

[0101] Where, λ U ≥0 is a hyperparameter that balances the contribution of the original training objective with the contribution of the weighted uniformity. This is achieved by applying equation (7) to... Joint optimization yields an optimal set of weight coefficients, which can significantly contribute to further compression efficiency. Furthermore, the weight unification loss takes into account how the convolution operation is performed as a GEMM matrix multiplication process at the underlying level, resulting in optimized weight coefficients that can significantly accelerate computation. Notably, the weight unification loss can be viewed as... R >0) or not have (when λ) R When the regularization is 0, an additional regularization term is added to the target loss. Furthermore, this method can be flexibly applied to arbitrary regularization losses.

[0102] In the embodiment, the weighted uniform loss is £. U (Θ) further includes compression ratio loss ￡ C (Θ), Unified Distortion Loss I (Θ) and calculation of velocity loss ￡ S (Θ):

[0103] £ U (Θ)=￡ I (Θ)+λ C £ C (Θ)+λ S £ S (Θ), (8)

[0104] Detailed explanations of these loss terms will be provided in later chapters. For better learning performance and efficiency, an iterative optimization algorithm is executed. In the first step, the partial weight coefficients that satisfy the desired structure are fixed. Then, in the second step, the unfixed parts of the weight coefficients are updated by training the loss through backpropagation. By iteratively performing these two steps, more and more weights can be gradually fixed, and the joint loss can be effectively optimized incrementally.

[0105] Furthermore, in the embodiments, each layer is compressed individually, therefore It can be further written as:

[0106]

[0107] Among them, L U (W j ) is the uniform loss defined at the j-th layer; N is the total number of layers containing the measurement quantization loss; W j Let L represent the weight coefficient of the j-th layer. Again, since L is calculated independently for each layer... U (W j Therefore, in the remainder of this disclosure, the superscript j may be omitted without loss of generality.

[0108] For each network layer, its weight coefficients W have a size (c i ,k1,k2,k3,c o The layer is a 5-dimensional (5D) tensor of size (h). i ,w i ,d i ,c i The layer has a 4-dimensional (4D) tensor A, and the output of this layer is a tensor of size (h). o ,w o ,d o ,c o The four-dimensional tensor B has size c. i k1, k2, k3, c o h i w i d i h o w o d o It is an integer greater than or equal to 1. When the size c i k1, k2, k3, c o h i w i d i h o w o d oWhen any one of the terms in M ​​is 1, the corresponding tensor is reduced to a lower dimension. Each term in each tensor is a floating-point number. Let M represent a 5D binary mask of the same size as W, where each term in M ​​is a binary number 0 / 1, indicating whether the corresponding weight coefficient is pruned / remains unchanged. M is introduced in association with W to address the case where W comes from a pruned DNN model, in which some connections between neurons in the network are removed from computation. When W comes from the original, unpruned, pre-trained model, all terms in M ​​are 1. Output B is computed via a convolution operation ⊙ based on A, M, and W:

[0109]

[0110] l = 1, ..., h i m=1,…,w i n=1,…,d i ,l′=1,…,h o ,

[0111] m′=1,…,w o ,n′=1,…,d o v = 1, ..., c o (10)

[0112] parameter h i w i and d i (h0、w o and d o ) represents the height, weights, and depth of the input tensor A (output tensor B). Parameter c i (c o ) represents the number of input (output) channels. Parameters k1, k2, and k3 correspond to the sizes of the convolution kernels along the height, weight, and depth axes, respectively. That is, for each output channel v = 1, ..., c o The operation described in equation (10) can be viewed as a convolution with input A of size (c i The 4D weighted tensor W of k1, k2, k3) v .

[0113] The order of the summation operations in equation (10) can be changed, and in the embodiment, the operation of equation (10) is performed as follows: The 5D weight tensor is reshaped to a size of (c′) i ,c′ o A 2D matrix, where c′ i ×c′ o =c i ×c o ×k1×k2×k3. For example, in some embodiments, it is c′. i=c i ,c′ o =c o ×k1×k2×k3 or c′ o =c o ,c′ i =c i ×k1×k2×k3.

[0114] The desired structure of the weight coefficients is designed by considering the following two aspects. First, the structure of the weight coefficients is consistent with how the underlying GEMM matrix multiplication process of convolution operations is implemented, thereby accelerating the speed of inference computation using the learned weight coefficients. Second, the structure of the weight coefficients can help improve quantization and entropy coding efficiency for further compression. In the embodiment, a block-by-block structure for the weight coefficients is used in each layer of the 2D reshaped weight matrix. Specifically, the 2D matrix is ​​divided into blocks of size (g i ,g o The weights within a block are uniformized. The uniformized weights are set to follow predefined uniformity rules, such as setting all values ​​to be the same so that a single value can be used to represent the entire block in a highly efficient quantization process. Multiple rules for uniformizing weights can exist, each associated with a uniform distortion loss, which measures the error introduced by applying that rule. For example, instead of setting weights to have the same absolute value while preserving their original positive or negative sign, the weights could be set to be identical. Given this design structure, during iteration, the portion of the weight coefficients to be fixed is determined by taking into account the uniform distortion loss, the estimated compression ratio loss, and the estimated speed loss. Then, in a second step, the normal neural network training process is performed, and the remaining unfixed weight coefficients are updated via backpropagation.

[0115] Figure 4C The overall framework of the iterative retraining / fine-tuning process is shown, which iteratively alternates between the two steps to gradually optimize the joint loss of equation (7). Given a pre-trained DNN model with weight coefficients W and a mask M (which can be a pruned sparse model or an unpruned non-sparse model), in the first step, the reshaping module 440 determines the weight unification method u* through a Unification Method Selection process. In this process, the reshaping module 440 reshapes the weight coefficients W (and the corresponding mask M) into a size (c′). i ,c′ o The 2D weight matrix W is then reshaped into a 2D matrix of size (g). i ,g oThe weights in block b are uniform. Weight unification occurs within each block. For each block b, a weight unifier is used to unify the weight coefficients within the block. Different methods can exist to unify the weight coefficients in b. For example, the weight unifier can set all weights in b to the same value, such as setting it to the average of all weights in b. In this case, the L of the weight coefficients in b... N Norms (e.g., the L2 norm as the weighted variance in b) reflect the uniform distortion loss when using the mean to represent the entire block. I (b). Furthermore, the weight unifier can set all weights to have the same absolute value while preserving their original positive or negative sign. In this case, L is the absolute value of the weights in b. N Norm can be used to measure L I (b) In other words, given a weight unification method u, the weight unifier can use method u in combination with the associated unification distortion loss L. I (u,b) is used to unify the weights in b.

[0116] Similarly, the compressibility loss ε in equation (8) C (u,b) reflects the compression efficiency of using method u to unify the weights in b. For example, when all weights are set to the same value, using only one number to represent the entire block results in a compression ratio of r. 压缩 =g i ·g o £ C (u,b) can be defined as 1 / r 压缩 .

[0117] The velocity loss in equation (8) S (u,b) reflects the estimated computational speed using the uniform weighting coefficients in b, which are obtained using method u. The estimated computational speed is a function of the number of multiplication operations in the computation using the uniform weighting coefficients.

[0118] So far, for every possible method u of unifying the weights in b using a weight unifier, based on ￡ I (u,b), £ C (u,b), £ S (u,b) Calculate the weighted uniform loss ￡ in equation (8). U (u,b). A uniform loss with minimum weights can be used. U *(u,b) is used to select the optimal weight unification method u*.

[0119] Once the weight unification method u* is determined for each block b, the objective becomes to find a set of updated optimal weight coefficients W* and corresponding weight masks M* by iteratively minimizing the joint loss described in equation (7). Specifically, for the t-th iteration, there exists a current weight coefficient W(t-1) and mask M(t-1). Furthermore, the weight unifying mask Q(t-1) is maintained throughout the training process. The weight unifying mask Q(t-1) has the same shape as W(t-1), thus recording whether the corresponding weight coefficients have been unified. Then, the weight unification module 445 calculates the unified weight coefficients W* through the weight unification process. U (t-1) and the new unified mask Q(t-1). During weight unification, these blocks are based on their unified loss ε. U (u*,b) are sorted in ascending order. Given the hyperparameter q, the top q% of blocks are selected for unification. The weight unifier uses a corresponding determined method u* to unify the blocks in the selected block b, thereby producing a unified weight W. U (t-1) and weight mask M U (t-1). The corresponding entries in the unified mask Q(t-1) are marked as unified. In the embodiment, M U Unlike M(t-1), in this case, for a block that has both pruned and uncrunted weight coefficients, the original pruned weight coefficients will be reset to non-zero values ​​by the weight unifier, and M... U The corresponding item in (t-1) will be changed. In another embodiment, M U (t-1) is the same as M(t-1). In this case, for a block that has both pruned and unpruned weight coefficients, only the unpruned weights will be reset, while the pruned weights will remain at zero.

[0120] Then, in the second step, the weight update module 450 fixes the weight coefficients marked as unified by Q(t-1), and then updates the remaining unfixed weight coefficients in W(t-1) through the neural network training process, thereby generating updated W(t) and M(t).

[0121] set up Let represent the training dataset, where Can be compared with the original dataset Same, based on Obtain the pre-trained weight coefficients W. It can also be with Different datasets, but with similarities to the original dataset. The same data distribution. In the second step, the network forward computation module 415 uses the current weight coefficients W. U (t-1) and mask M U (t-1) allows each input x to pass through the current network via the network's forward computation process, thereby generating an estimated output. Based on the truth label y and the estimated output The target loss calculation module 420 calculates the target training loss in equation (7) through the target loss calculation process. Then, the gradient calculation module 425 calculates the gradient G(W) of the target loss. U (t-1)). The automatic gradient calculation methods used by deep learning frameworks such as TensorFlow or PyTorch can be used to calculate G(W). U (t-1)). Based on gradient G(W) U (t-1)) and the unified mask Q(t-1), the weight update module 450 uses backpropagation and the weight update process to update W through backpropagation. U The unfixed weighting coefficients of (t-1) and the corresponding mask M U Update W at (t-1). The retraining process itself is also an iterative process. Multiple iterations are used to update W. U The unfixed portions of (t-1) and the corresponding M(t-1) are updated, for example, until the target loss converges. Then, the system proceeds to the next iteration t, given the new hyperparameters q(t), based on W. U (t-1) and u* can be used to calculate the new unified weight coefficient W through the weight unification process. U (t-1), Mask M U (t) and the corresponding unified mask Q(t).

[0122] In the embodiment, the hyperparameter q(t) increases its value as t increases during each iteration, so that more and more weight coefficients will be unified and fixed throughout the iterative learning process.

[0123] The goal of unified regularization is to improve the efficiency of further compressing the learned weight coefficients, thereby accelerating computation by using optimized weight coefficients. This can significantly reduce the size of the DNN model and speed up inference computation.

[0124] Through an iterative retraining process, this method effectively maintains the performance of the original training objective while pursuing compression and computational efficiency. The iterative retraining process also provides the flexibility to introduce different losses at different times, allowing the system to focus on different objectives during optimization.

[0125] This method can be applied to datasets with different data formats. The input / output data are 4D tensors, which can be real video clips, images, or extracted feature maps.

[0126] Figure 4D This is a functional block diagram of a training device 400D for compressing a neural network model using microstructured weight pruning and weight unification, according to some other embodiments. Figure 4E This is a functional block diagram of a training device 400E for compressing a neural network model using microstructured weight pruning and weight unification, according to some other embodiments.

[0127] like Figure 4D As shown, the training device 400D includes a microstructure selection module 455, a weight pruning / unification module 460, a network forward calculation module 415, a target loss calculation module 420, a gradient calculation module 425, and a weight update module 465.

[0128] like Figure 4E As shown, the training device 400E includes a microstructure selection module 455, a weight pruning / unification module 460, a network forward computation module 415, a target loss computation module 420, a gradient computation module 425, and a weight update module 465. The training device 400E further includes a mask computation module 470.

[0129] From another perspective, the pre-trained weight coefficients Θ can be further trained through another network process, during which the optimal set of weight coefficients can be learned to improve the efficiency of further model compression and inference acceleration. This disclosure describes a microstructured pruning and unified method for achieving this goal.

[0130] Specifically, the microstructured weight pruning loss is defined. Unified loss with microstructured weights They are optimized together with the original target loss:

[0131]

[0132] Where, λ S ≥0 and λ U ≥0 is a parameter used to balance the contributions of the original training objective, the weight unification objective, and the weight pruning objective. This is achieved by applying equation (11) to... Joint optimization yields an optimal set of weight coefficients, which can significantly contribute to the effectiveness of further compression. Furthermore, the weight unification loss takes into account how the convolution operation is performed as a GEMM matrix multiplication process at the underlying level, resulting in optimized weight coefficients that can significantly accelerate computation. It is worth noting that the weight pruning loss and the weight unification loss can be viewed as... (when λ...) R >0) or not having (when λ) R In the case of 0) regularization, an additional regularization term is added to the target loss. Furthermore, this method can be flexibly applied to arbitrary regularization losses. R (Θ).

[0133] To improve learning efficiency, an iterative optimization algorithm is executed. In the first step, some weight coefficients that satisfy the desired structure are fixed. Then, in the second step, the unfixed parts of the weight coefficients are updated by training the loss through backpropagation. By iteratively performing these two steps, more and more weights can be gradually fixed, and the joint loss can be effectively optimized incrementally.

[0134] Furthermore, in the embodiments, each layer is compressed individually, therefore and It can be further written as:

[0135]

[0136] Among them, L U (W j ) is the uniform loss defined at the j-th layer; L s (W j ) is the pruning loss defined at the j-th layer, N is the total number of layers involved in the training process, and W j Let L represent the weight coefficient of the j-th layer. Again, since L is calculated independently for each layer... U (W j ) and L S (W j In the remainder of this disclosure, the superscript j is omitted without loss of generality.

[0137] For each network layer, its weight coefficients W have a size (c i ,k1,k2,k3,c o The layer is a 5-dimensional (5D) tensor of size (h). i ,w i ,d i ,c i The layer has a 4-dimensional (4D) tensor A, and the output of this layer is a tensor of size (h).o ,w o ,d o ,c o The four-dimensional tensor B has size c. i k1, k2, k3, c o h i w i d i h o w o d o It is an integer greater than or equal to 1. When the size c i k1, k2, k3, c o h i w i d i h o w o d o When any term in M ​​is 1, the corresponding tensor is reduced to a lower dimension. Each term in each tensor is a floating-point number. Let M represent a 5D binary mask of the same size as W, where each term in M ​​is a binary number 0 / 1, indicating whether the corresponding weight coefficient has been pruned or remains unchanged during pre-pruning. M is introduced in association with W to address the case where W comes from a pruned DNN model, in which some connections between neurons in the network have been removed from computation. When W comes from the original uncrunted dense model, all terms in M ​​are 1. Output B is computed via convolution operations ⊙ based on A, M, and W:

[0138]

[0139] l = 1, ..., h i m=1,…,w i n=1,…,d i ,l′=1,…,h o ,

[0140] m′=1,…,w o ,n′=1,…,d o v = 1, ..., c o (13)

[0141] parameter h i w i and d i (h0、w o and d o ) represents the height, weights, and depth of the input tensor A (output tensor B). Parameter c i (c o) represents the number of input (output) channels. Parameters k1, k2, and k3 correspond to the sizes of the convolution kernels along the height, weight, and depth axes, respectively. That is, for each output channel v = 1, ..., c o The operation described in equation (13) can be viewed as a convolution with input A of size (c i The 4D weighted tensor W of k1, k2, k3) v .

[0142] The order of the summation operations in equation (13) can be changed to produce different configurations of the shapes of the input A, weights W (and mask M) to obtain the same output B. In the embodiment, two configurations are used. (1) The 5D weight tensor is reshaped to a size of (c′ i ,c′ o A 3D tensor of (k), where c′ i ×c′ o ×k=c i ×c o ×k1×k2×k3. For example, this configuration is c′. i =c i ,c′ o =c o ,k=k1×k2×k3. (2) Reshape the 5D weight tensor to a size of (c′ i ,c′ o A 2D matrix, where c′ i ×c′ o =c i ×c o ×k1×k2×k3. For example, some configurations are c′. i =c i ,c′ o =c o ×k1×k2×k3 or c′ o =c o ,c′ i =c i ×k1×k2×k3.

[0143] The desired microstructure of the weight coefficients is designed by considering the following two aspects. First, the microstructure of the weight coefficients is consistent with how the underlying GEMM matrix multiplication process of convolution operations is implemented, thereby accelerating the speed of inference computation using the learned weight coefficients. Second, the microstructure of the weight coefficients can help improve quantization and entropy coding efficiency for further compression. In the embodiment, a block-by-block structure for the weight coefficients is used in each layer of the 3D reshaped weight tensor or the 2D reshaped weight matrix. Specifically, in the case of the reshaped 3D weight tensor, it is divided into layers of size (g i ,g o ,gk The reshaped 2D weight matrix is ​​divided into blocks of size (g), and all coefficients within the blocks are pruned or uniform. i ,g o The block is defined as follows: all coefficients within the block are either pruned or unified. The pruned weights in the block are set to all zero. A pruning loss can be calculated for the block, which measures the error introduced by such pruning operations. Unified weights in the block are set to follow predefined unification rules, such as setting all values ​​to be the same so that a single value can be used to represent the entire block in a highly efficient quantization process. Multiple rules for unifying weights can exist, each associated with a unification distortion loss, which measures the error introduced by applying that rule. For example, instead of setting weights to have the same absolute value and preserving their original sign, given this microstructure, the parts of the weight coefficients to be pruned or unified are determined during iteration by taking into account both the pruning and unification losses. Then, in a second step, the pruned and unified weights are fixed, the normal neural network training process is performed, and the remaining unfixed weight coefficients are updated via backpropagation.

[0144] Figure 4D and 4E These are two embodiments of the iterative retraining / fine-tuning process, both iteratively alternating between two steps to progressively optimize the joint loss of equation (11). Given a pre-trained DNN model with weight coefficients {W} and a mask {M} (which can be a pruned sparse model or an unpruned non-sparse model), in the first step, both embodiments first reshape the weight coefficients W (and the corresponding mask M) of each layer into the desired 3D tensor or 2D matrix. Then, for each layer, the microstructure selection module 455 determines a set of pruned microstructures {b} whose weights will be pruned through a pruning and unification micro-structure selection process. s} or PMB, and determine the weights to be unified in a set of unified microstructures {b u} or Unification micro-structure block (UMB). Determine the pruned microstructure {b s} and unified microstructure {b uThere are several methods for this, and four are listed here. In method 1, for each layer with weight coefficients W and a mask M, for each block b in W, a weight unifier is used to unify the weight coefficients within the block (e.g., by setting all weights to have the same absolute value while preserving their original positive or negative signs). The corresponding unification loss L is then calculated. u (b) to measure uniform distortion (e.g., L of the absolute value of the weights in b). N Norm). Unified loss L u (W) can be calculated as L of all blocks in W. u (b) sum. Based on this unified loss L u (W), all layers of the DNN model are based on L u (W) Sort in ascending order. Then, given a uniform ratio u, the microstructural blocks (i.e., {b}) are selected. u The top layers (including all blocks in the selected layer) will be unified such that the actual unification ratio u′ (measured by the ratio of the total number of unified microstructure blocks in the selected layer to the total number of microstructure blocks in the entire DNN model) is closest to but still less than u%. Then, for each of the remaining layers, for each microstructure block b, the pruning loss L is calculated. s (b) (e.g., the sum of the absolute values ​​of the weights in b). Given a pruning ratio p, the blocks of this layer are determined according to L. s (b) Sort in ascending order and select the top p% of blocks as the {b} to be pruned. s For the remaining blocks of this layer, an optional additional step can be taken, in which the remaining blocks of this layer are based on the uniform loss L. u (b) Sort in ascending order and select the top (uu′)% as the {b} to be unified. u}

[0145] In Method 2, for each layer with weighting coefficients W and a mask M, the uniform loss L is calculated in a manner similar to that in Method 1. u (b) and L u (W). Then, given a uniform ratio u, the microstructural blocks of the first layer are uniformized using a method similar to Method 1. Then, the pruning loss L of the remaining layers is calculated using the same method. s (b) Given a pruning ratio p, all blocks in all remaining layers according to L s (b) Sort the blocks in ascending order and prune the top p% of the blocks. For the remaining blocks in the remaining layers, an optional additional step can be taken, in which the remaining blocks in the remaining layers are pruned based on a uniform loss L. u(b) Sort in ascending order and select the top (uu′)% as the {b} to be unified. u}

[0146] In Method 3, for each layer with weighting coefficients W and a mask M, for each block b in W, the uniform loss L is calculated in a manner similar to that in Method 1. u (b) and pruning loss L s (b). Given a pruning ratio p and a uniform ratio u, the blocks of this layer are based on L. s (b) Sort in ascending order and select the top p% of blocks as the {b} to be pruned. s For the remaining blocks in this layer, according to the uniform loss L... u (b) Sort in ascending order, then select the first u% as the {b} to be unified. u}

[0147] In Method 4, for each layer with weight coefficients W and a mask M, for each block b in W, the uniform loss L is calculated in a manner similar to that in Method 1. u (b) and pruning loss L s (b). Given a pruning ratio p and a uniform ratio u, all blocks in all layers of the DNN model are categorized according to L. s (b) Sort the blocks in ascending order and prune the top p% of the blocks. For the remaining blocks in the entire model, apply the uniform loss L... u (b) Sort in ascending order, and then select the first u% to unify.

[0148] After obtaining the set of pruned microstructures and the set of unified microstructures, the objective becomes to find a set of updated optimal weight coefficients W* and corresponding weight masks M* by iteratively minimizing the joint loss described in equation (11). Figure 4D In the first embodiment shown, for the t-th iteration, there exists a current weight coefficient W(t-1). Furthermore, a microstructured uniform mask U(t-1) and a microstructured pruning mask P(t-1) are maintained throughout the training process. Both U(t-1) and P(t-1) have the same shape as W(t-1), thereby recording whether the corresponding weight coefficients are uniform or pruned, respectively. Then, the weight pruning / unification module 460 calculates the pruned and uniform weight coefficient W through the weight pruning and unification process. PU (t-1), in this process, the selected pruned microstructure masked by P(t-1) is pruned, and the masked weights in the selected unified microstructure are unified through U(t-1), thereby generating an updated weight mask M. PU (t-1). In the embodiment, M PU(t-1) Unlike the pre-training pruning mask M, in this case, for blocks that simultaneously have pre-pruned and non-pre-pruned weight coefficients, the original pruned weight coefficients will be reset to non-zero values ​​by the weight unifier, and M... PU The corresponding item in (t-1) will be changed. In another embodiment, M PU (t-1) is the same as M. In this case, for a block that has both pruned and unpruned weight coefficients, only the unpruned weights will be reset, while the pruned weights will remain at zero.

[0149] Then, in the second step, the weight update module 465 fixes the weight coefficients marked by U(t-1) and P(t-1) as microstructured unified or microstructured pruned. Then, the remaining unfixed weight coefficients in W(t-1) are updated through the neural network training process, thereby generating updated W(t) and M(t).

[0150] Specifically, set Let represent the training dataset, where Can be compared with the original dataset Same, based on Obtain the pre-trained weight coefficients W. It can also be with Different datasets, but with similarities to the original dataset. The same data distribution. In the second step, the network forward computation module 415 uses the current weight coefficients W. U (t-1) and a mask M are used to pass each input x through the current network via the network's forward computation process, thereby generating an estimated output. Based on the truth label y and the estimated output The target loss calculation module 420 calculates the target training loss in equation (11) through the target loss calculation process. Then, the gradient calculation module 425 calculates the gradient G(W) of the target loss. U (t-1)). The automatic gradient calculation methods used by deep learning frameworks such as TensorFlow or PyTorch can be used to calculate G(W). U (t-1)). Based on gradient G(W) U (t-1)) and microstructured unified mask U(t-1) and microstructured pruning mask P(t-1), weight update module 465 uses backpropagation and weight update process to update W through backpropagation. U The unfixed weight coefficients of (t-1) are updated. The retraining process itself is also an iterative process. Multiple iterations are used to update W. UThe unfixed portion of (t-1) is updated, for example, until the target loss converges. Then, the system proceeds to the next iteration t, in which, given a new uniform scaling u(t) and pruning scaling p(t), a new set of uniform and pruned microstructures (as well as new microstructured uniform masks U(t) and microstructured pruning masks P(t)) are determined through a pruning and uniform microstructure selection process.

[0151] exist Figure 4E In the second embodiment of the training process shown, a set of updated optimal weight coefficients W* and corresponding weight masks M* are found through another iterative process. For the t-th iteration, there exists a current weight coefficient W(t-1) and mask M. Furthermore, the mask calculation module 470 calculates the microstructured unified mask U(t-1) and the microstructured pruned mask P(t-1) through a mask and unified pruning calculation process. U(t-1) and P(t-1) both have the same shape as W(t-1), thereby recording whether the corresponding weight coefficients are unified or pruned, respectively. Then, the weight pruning / unification module 460 calculates the pruned and unified weight coefficients W* through a weight pruning and unification process. PU (t-1), in this process, the selected pruned microstructure masked by P(t-1) is pruned, and the masked weights in the selected unified microstructure are unified through U(t-1), thereby generating an updated weight mask M. PU (t-1).

[0152] Then, in the second step, the weight update module 465 fixes the weight coefficients marked by U(t-1) and P(t-1) as microstructured unified or microstructured pruned. Then, it updates the remaining unfixed weight coefficients in W(t-1) through the neural network training process, thereby generating the updated W(t). Similar to... Figure 4D The first embodiment, given a training dataset In this case, the network forward computation module 415 uses the current weight coefficients W(t-1) and mask M(t-1) to pass each input x through the current network via the network forward computation process, thereby generating an estimated output. Based on the truth label y and the estimated output The target loss calculation module 420 calculates the joint training loss through the joint loss calculation process. The joint training loss includes the target training loss in equation (11). and residual loss res (W(t-1)), as shown in equation (5).

[0153] £ res(W(t-1)) measures the current weight W(t-1) and the target weight W that has been pruned and unified. PU The difference between (t-1). For example, the L1 norm can be used:

[0154] £ res (W(t-1))=||W(t-1))-W PU (t-1)|| (14)

[0155] Then, the gradient calculation module 425 calculates the gradient of the joint loss G(W(t-1)). Automatic gradient calculation methods used by deep learning frameworks such as TensorFlow or PyTorch can be used to calculate G(W(t-1)). Based on the gradient G(W(t-1)) and the microstructured unified mask U(t-1) and the microstructured pruning mask P(t-1), the weight update module 465 updates the unfixed weight coefficients of W(t-1) through backpropagation using a backpropagation and weight update process. The retraining process itself is also an iterative process. Multiple iterations are used to update the unfixed portion of W(t-1), for example, until the target loss converges. Then, the system proceeds to the next iteration t, in which, given the unified proportion u(t) and the pruning proportion p(t), a new set of unified and pruned microstructures (as well as new microstructured unified masks U(t) and microstructured pruning masks P(t)) are determined through a pruning and unified microstructure selection process.

[0156] During the entire iteration process, at the T-th iteration, the pruned and unified weight coefficients can be calculated through a weight pruning and unification process. In this process, the selected pruned microstructure masked by P(T) is pruned, and the masked weights in the selected unified microstructure are unified through U(T), thereby generating an updated weight mask M. PU (T). Similar to the previous one. Figure 4D In an embodiment, M PU (T) can be the same as the mask M before pruning. In this case, for a block that has both pruned and uncropped weight coefficients, the original pruned weight coefficients will be reset to non-zero values ​​by the weight unifier, and M... PU The corresponding item in (T) will be changed. Furthermore, M PU (T) can be the same as M. In this case, for a block that has both pruned and undrained weight coefficients, only the undrained weights will be reset, while the pruned weights will remain at zero. W PU (T) and M PU (T) can be used to generate the final updated models W* and M*. For example, W* = W PU (T) and M*=M·MPU (T).

[0157] In the embodiment, the hyperparameters u(t) and p(t) can increase their respective values ​​as t increases during iteration, so that more and more weight coefficients will be pruned, unified and fixed throughout the iterative learning process.

[0158] The goal of unified regularization is to improve the efficiency of further compressing the learned weight coefficients, thereby accelerating computation by using optimized weight coefficients. This can significantly reduce the model size of DNNs and speed up inference computation.

[0159] Through an iterative retraining process, this method effectively maintains the performance of the original training objective while pursuing compression and computational efficiency. The iterative retraining process also provides the flexibility to introduce different losses at different times, allowing the system to focus on different objectives during the optimization process.

[0160] This method can be applied to datasets with different data formats. The input / output data are 4D tensors, which can be real video clips, images, or extracted feature maps.

[0161] Figure 5 This is a flowchart of a method 500 for training a neural network model compression employing microstructured weight pruning and weight unification, according to an embodiment.

[0162] In some implementations, Figure 5 One or more process blocks can be executed by platform 120. In some implementations, Figure 5 One or more process blocks can be executed by another device or another group of devices (such as user equipment 110) that is separate from or includes platform 120.

[0163] Method 500 is executed to train a deep neural network, which is used to reduce the parameters of the input neural network to obtain an output neural network.

[0164] like Figure 5 As shown, in operation 510, method 500 includes selecting a pruning microstructure block to be pruned from a plurality of blocks of input weights of a deep neural network that are masked by an input mask.

[0165] In operation 520, method 500 includes pruning the input weights based on selected pruning microstructure blocks.

[0166] In operation 530, method 500 includes updating the input mask and the pruning mask based on the selected pruning microstructure block, the pruning mask indicating whether each of the input weights has been pruned.

[0167] In operation 540, method 500 includes updating the pruned input weights and the updated input mask based on the updated pruned mask to minimize the loss of the deep neural network.

[0168] The updating of the pruned input weights and the updated input mask may include: using a deep neural network to reduce the parameters of a first trained neural network to estimate a second trained neural network whose input weights are unified and masked by the updated input mask; determining the loss of the deep neural network based on the estimated second trained neural network and the ground-truth neural network; determining the gradient of the determined loss based on the pruned input weights; and updating the pruned input weights and the updated input mask based on the determined gradient and the updated pruned mask to minimize the determined loss.

[0169] Deep neural networks can be further trained through the following steps: reshaping the input weights masked by the input mask; dividing the reshaping input weights into multiple blocks of input weights; unifying multiple weights in one or more blocks of the input weights divided by the reshaping input weights; updating the input mask and the unifying mask based on the unified multiple weights, the unifying mask indicating whether each of the input weights has been unified; and updating the updated input mask and input weights based on the updated unifying mask to minimize the loss of the deep neural network, where multiple weights in one or more blocks of the input weights have been unified.

[0170] Updating the updated input mask and input weights may include: using a deep neural network to reduce the parameters of a first trained neural network to estimate a second trained neural network whose input weights are unified and masked by the updated input mask; determining the loss of the deep neural network based on the estimated second trained neural network and the ground truth neural network; determining the gradient of the determined loss based on the input weights, where multiple weights in one or more blocks of a plurality of blocks are unified; and updating the pruned input weights and the updated input mask based on the determined gradient and the updated unified mask to minimize the determined loss.

[0171] Deep neural networks can be further trained by: selecting a unified microstructure block to be unified from multiple blocks of input weights masked by an input mask; unifying multiple weights in one or more blocks of pruned input weights based on the selected unified microstructure block to obtain pruned and unified input weights for the deep neural network; and updating the unified mask based on the unified weights, the unified mask indicating whether each of the input weights has been unified. Updating the input mask may include: updating the input mask based on the selected pruned microstructure block and the selected unified microstructure block to obtain a pruned unified mask. Updating the pruned input weights and the updated input mask may include: updating the pruned and unified input weights and the pruned unified mask based on the updated pruned mask and the updated unified mask to minimize the loss of the deep neural network.

[0172] Updating the pruned and unified input weights and the pruning-unified mask may include: using a deep neural network to reduce the parameters of a first trained neural network to estimate a second trained neural network whose pruned and unified input weights are masked by the pruning-unified mask; determining the loss of the deep neural network based on the estimated second trained neural network and the ground truth neural network; determining the gradient of the determined loss based on the input weights, where multiple weights in one or more blocks of a plurality of blocks are unified; and updating the pruned and unified input weights and the pruning-unified mask based on the determined gradient, the updated pruning mask, and the updated unified mask to minimize the determined loss.

[0173] The pruning microstructure block can be selected from multiple blocks of input weights masked by the input mask based on a predetermined pruning ratio for the input weight to be pruned in each iteration.

[0174] Figure 6 This is a diagram of an apparatus 600 for training a neural network model compression employing microstructured weight pruning and weight unification, according to an embodiment.

[0175] like Figure 6 As shown, the device 600 includes selection code 610, pruning code 620, first update code 630, and second update code 640.

[0176] Device 600 trains a deep neural network, which is used to reduce the parameters of the input neural network to obtain an output neural network.

[0177] Selection code 610 is configured to enable at least one processor to select a pruning microstructure block to be pruned from a plurality of blocks of input weights of a deep neural network that are masked by an input mask.

[0178] Pruning code 620 is configured to enable at least one processor to prune the input weights based on selected pruning microstructure blocks.

[0179] The first update code 630 is configured to cause at least one processor to update the input mask and the pruning mask based on the selected pruning microstructure block, the pruning mask indicating whether each of the input weights has been pruned.

[0180] The second update code 640 is configured to cause at least one processor to update the pruned input weights and the updated input mask based on the updated pruned mask, in order to minimize the loss of the deep neural network.

[0181] The second update code 640 may be further configured to cause at least one processor to: reduce the parameters of a first training neural network using a deep neural network to estimate a second training neural network, the input weights of which are unified and masked by an updated input mask; determine the loss of the deep neural network based on the estimated second training neural network and the ground truth neural network; determine the gradient of the determined loss based on the pruned input weights; and update the pruned input weights and the updated input mask based on the determined gradient and the updated unified mask to minimize the determined loss.

[0182] Deep neural networks can be further trained through the following steps: reshaping the input weights masked by the input mask; dividing the reshaping input weights into multiple blocks of input weights; unifying multiple weights in one or more blocks of the input weights divided by the reshaping input weights; updating the input mask and the unifying mask based on the unified multiple weights, the unifying mask indicating whether each of the input weights has been unified; and updating the updated input mask and input weights based on the updated unifying mask to minimize the loss of the deep neural network, where multiple weights in one or more blocks of the input weights have been unified.

[0183] The second update code 640 may be further configured to cause at least one processor to: reduce the parameters of a first trained neural network using a deep neural network to estimate a second trained neural network, the input weights of which are unified and masked by an updated input mask; determine the loss of the deep neural network based on the estimated second trained neural network and the ground truth neural network; determine the gradient of the determined loss based on the input weights, in which multiple weights in one or more blocks of a plurality of blocks are unified; and update the pruned input weights and the updated input mask based on the determined gradient and the updated unified mask to minimize the determined loss.

[0184] Deep neural networks can be further trained through the following steps: selecting a unified microstructure block to be unified from multiple blocks of input weights masked by the input mask; unifying multiple weights in one or more blocks of pruned input weights based on the selected unified microstructure block to obtain pruned and unified input weights for the deep neural network; and updating the unified mask based on the unified weights, the unified mask indicating whether each of the input weights has been unified. Updating the input mask may include updating the input mask based on the selected pruned microstructure block and the selected unified microstructure block to obtain a pruned unified mask. Updating the pruned input weights and the updated input mask may include updating the pruned and unified input weights and the pruned unified mask based on the updated pruned mask and the updated unified mask to minimize the loss of the deep neural network.

[0185] The second update code 640 may be further configured to cause at least one processor to: reduce the parameters of a first trained neural network using a deep neural network to estimate a second trained neural network, the pruned and unified input weights of which are masked by a pruning-unification mask; determine the loss of the deep neural network based on the estimated second trained neural network and the ground truth neural network; determine the gradient of the determined loss based on the input weights, in which multiple weights in one or more blocks of a plurality of blocks are unified; and update the pruned and unified input weights and the pruning-unification mask based on the determined gradient, the updated pruning mask, and the updated unification mask to minimize the determined loss.

[0186] The pruning microstructure block can be selected from multiple blocks of input weights masked by the input mask based on a predetermined pruning ratio for the input weight to be pruned in each iteration.

[0187] The foregoing disclosure provides illustrations and descriptions, but is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed. Modifications and variations can be made based on the foregoing disclosure, or modifications and variations can be obtained from the actual operation of the embodiments.

[0188] As used herein, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

[0189] It is evident that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or combinations of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, this document describes the operation and behavior of the systems and / or methods without referring to any specific software code—it is understood that software and hardware can be designed to implement systems and / or methods based on those described herein.

[0190] Although combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible embodiments. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not specifically disclosed in the specification. Although each dependent claim listed below may depend directly on only one claim, the disclosure of possible embodiments includes combinations of each dependent claim with every other claim in the claim set.

[0191] No element, action, or instruction used herein should be construed as critical or necessary unless explicitly described as such. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and may be used interchangeably with “one or more.” Additionally, as used herein, the term “set” is intended to include one or more items (e.g., related items, unrelated items, a combination of related and unrelated items, etc.) and may be used interchangeably with “one or more.” The term “an” or similar language is used if only one item is intended to be referred to. Furthermore, as used herein, the terms “including,” “having,” etc., are intended to indicate open-ended terms. Further, unless otherwise explicitly stated, the word “based on” is intended to mean “at least partially based on.”

Claims

1. A method for compressing neural network models, comprising: Receives input neural network and input mask; The parameters of the input neural network are reduced using a deep neural network, which is trained through the following steps: From a plurality of blocks of input weights of the deep neural network that are masked by the input mask, select the pruning microstructure block to be pruned; The input weights are pruned based on the selected pruned microstructure blocks; From a plurality of blocks of input weights masked by the input mask, select a unified microstructure block to be unified; as well as Based on the selected unified microstructure blocks, multiple weights in one or more blocks of pruned input weights are unified to obtain the pruned and unified input weights of the deep neural network. as well as Based on the pruned and unified input weights of the input neural network and the deep neural network, an output neural network with reduced parameters is obtained. The deep neural network is further trained through the following steps: Based on the unified weights, the input mask and the unified mask are updated, and the unified mask indicates whether each of the input weights has been unified. Based on the updated unified mask, the parameters of the first trained neural network are reduced using the deep neural network to estimate the second trained neural network, wherein the input weights of the deep neural network are unified and masked by the updated input mask. Based on the estimated second trained neural network and ground truth neural network, the loss of the deep neural network is determined; The gradient of the determined loss is determined based on the input weights, wherein the weights in one or more of the plurality of blocks are unified; and Based on the determined gradient and the updated unified mask, the pruned input weights and the updated input mask are updated to minimize the determined loss.

2. The method according to claim 1, wherein, The deep neural network is further trained through the following steps: Based on the selected pruned microstructure block, the input mask and the pruned mask are updated, wherein the pruned mask indicates whether each of the input weights has been pruned; as well as Based on the updated pruning mask, the pruned input weights and the updated input mask are updated to minimize the loss of the deep neural network.

3. The method according to claim 1, wherein, The unified weights are obtained through the following steps: Reshape the input weights that are masked by the input mask; The reshaped input weights are divided into multiple blocks of the input weights; The weights in one or more blocks divided by the reshaped input weights are unified.

4. The method according to claim 2, wherein, The deep neural network is further trained through the following steps: updating a unified mask based on the unified weights in one or more of the multiple blocks, the unified mask indicating whether each of the input weights has been unified. Updating the input mask includes: updating the input mask based on the selected pruned microstructure block and the selected unified microstructure block to obtain a pruned unified mask, and Updating the pruned input weights and the updated input mask includes updating the pruned and unified input weights and the pruned unified mask based on the updated pruned mask and the updated unified mask, so as to minimize the loss of the deep neural network.

5. The method according to claim 4, wherein, The update of the pruned and unified input weights and the pruned unified mask includes: Using the deep neural network, the parameters of the first trained neural network are reduced to estimate the second trained neural network, wherein the pruned and unified input weights of the deep neural network are masked by the pruned and unified mask. The loss of the deep neural network is determined based on the estimated second trained neural network and ground truth neural network; The gradient of the determined loss is determined based on the input weights, wherein the weights in one or more of the plurality of blocks are unified; and Based on the determined gradient, the updated pruning mask, and the updated unified mask, the pruned and unified input weights and the pruning unified mask are updated to minimize the determined loss.

6. The method according to any one of claims 1 to 5, wherein, Based on a predetermined pruning ratio for the input weights to be pruned in each iteration, the pruned microstructure block is selected from a plurality of blocks of input weights masked by the input mask.

7. An apparatus for compressing neural network models, the apparatus comprising: At least one memory configured to store program code; as well as At least one processor configured to read the program code and operate according to the instructions of the program code, the program code comprising: A receiving code, the receiving code being configured to cause the at least one processor to receive an input neural network and an input mask; Reduce code, the code reduction being configured to cause the at least one processor to reduce the parameters of the input neural network using a deep neural network trained through the following steps: From a plurality of blocks of input weights of the deep neural network that are masked by the input mask, select the pruning microstructure block to be pruned; The input weights are pruned based on the selected pruned microstructure blocks; From a plurality of blocks of input weights masked by the input mask, select the unified microstructure block to be unified; and Based on the selected unified microstructure blocks, multiple weights in one or more blocks of pruned input weights are unified to obtain the pruned and unified input weights of the deep neural network; and The acquisition code is configured to cause the at least one processor to output an output neural network with reduced parameters based on the pruned and unified input weights of the input neural network and the deep neural network. The code reduction is configured to enable the deep neural network to be further trained through the following steps: updating the input mask and the unified mask based on a plurality of unified weights, the unified mask indicating whether each of the input weights is unified; using the deep neural network, reducing the parameters of a first training neural network to estimate a second training neural network based on the updated unified mask, the input weights of the deep neural network being unified and masked by the updated input mask; determining the loss of the deep neural network based on the estimated second training neural network and the ground truth neural network; determining the gradient of the determined loss based on the input weights, where the plurality of weights in one or more of the plurality of blocks are unified; and updating the pruned input weights and the updated input mask based on the determined gradient and the updated unified mask to minimize the determined loss.

8. The apparatus according to claim 7, wherein, The deep neural network is further trained through the following steps: Based on the selected pruned microstructure block, the input mask and the pruned mask are updated, wherein the pruned mask indicates whether each of the input weights has been pruned; as well as Based on the updated pruning mask, the pruned input weights and the updated input mask are updated to minimize the loss of the deep neural network.

9. The apparatus according to claim 7, wherein, The unified weights are obtained through the following steps: Reshape the input weights that are masked by the input mask; The reshaped input weights are divided into multiple blocks of the input weights; The weights in one or more blocks divided by the reshaped input weights are unified.

10. The apparatus according to claim 8, wherein, The deep neural network is further trained through the following steps: updating a unified mask based on the unified weights in one or more of the multiple blocks, the unified mask indicating whether each of the input weights has been unified. Updating the input mask includes: updating the input mask based on the selected pruned microstructure block and the selected unified microstructure block to obtain a pruned unified mask, and Updating the pruned input weights and the updated input mask includes updating the pruned and unified input weights and the pruned unified mask based on the updated pruned mask and the updated unified mask, so as to minimize the loss of the deep neural network.

11. The apparatus according to claim 10, wherein, The update of the pruned and unified input weights and the pruned unified mask includes: Using the deep neural network, the parameters of the first trained neural network are reduced to estimate the second trained neural network, wherein the pruned and unified input weights of the deep neural network are masked by the pruned and unified mask. The loss of the deep neural network is determined based on the estimated second trained neural network and ground truth neural network; The gradient of the determined loss is determined based on the input weights, wherein the weights in one or more of the plurality of blocks are unified; and Based on the determined gradient, the updated pruning mask, and the updated unified mask, the pruned and unified input weights and the pruning unified mask are updated to minimize the determined loss.

12. The apparatus according to any one of claims 7 to 11, wherein, Based on a predetermined pruning ratio for the input weights to be pruned in each iteration, the pruned microstructure block is selected from a plurality of blocks of input weights masked by the input mask.

13. A non-transitory computer-readable medium storing instructions that, when executed by at least one processor for neural network model compression, cause the at least one processor to perform the method for neural network model compression according to any one of claims 1 to 6.

Images