Structured Pruning Method, System, and Computer-Readable Medium

Abstract

The present disclosure provides a structured pruning method, system and computer-readable medium. The structured pruning method includes: receiving an input weight pattern; obtaining an output weight pattern from the input weight pattern based on a block row and column pruning method, the block row and column pruning method including: dividing the input weight pattern into a mesh network of sub-patterns, each row of the input weight pattern including a first number of sub-rows of sub-patterns, and each column of the input weight pattern including a second number of sub-columns of sub-patterns; pruning corresponding one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and under the constraint that for each row of the input weight pattern, at least one sub-row of the sub-pattern in the corresponding row of the input weight pattern is not pruned, pruning corresponding one or more sub-rows from each sub-pattern to achieve a predetermined row sparsity. The embodiment of the present disclosure reduces the storage and computational costs associated with the weight pattern.

Description

Technical Field

[0001] The present disclosure relates to the field of neural network technology, and in particular to a structured pruning method, system, and computer-readable medium for machine learning models. Background Art

[0002] With the emergence and development of neural network technology, various types of neural network models have been explored and applied to many real-world applications, including speech recognition, natural language processing (NLP), prediction, data verification, risk management, etc. Given a specific application, such as speech recognition, a neural network model can be trained to learn rules and patterns related to the application from training samples. These learned rules and patterns are represented and stored in the form of weights (or strengths) of connections between nodes within and between layers of the neural network model, and are usually represented using a weight matrix.

[0003] For real-world applications, such weight matrices are typically very large in size and therefore occupy a large amount of memory, resulting in a large storage overhead. Moreover, when a neural network model is used to perform a prediction or decision (e.g., determining whether a sound is made by a certain person), a large number of multiplication and addition / accumulation operations need to be performed on the weight matrix to generate the output of such a prediction or decision. This will undoubtedly further lead to huge computational costs or overheads each time a prediction or decision is performed. These storage and computational overheads seriously affect the availability and performance of neural networks in terms of storage and time consumption, and may limit the use of neural networks in computing systems with high storage and computing capabilities. Summary of the Invention

[0004] This summary introduces a simplified concept of structured pruning, which will be further described in the detailed description below. This summary is not intended to identify essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] This disclosure describes exemplary embodiments for structured pruning of machine learning models.

[0006] In some embodiments, an input weight pattern of a machine learning model (e.g., a neural network model) may be received. A block-wise row and column pruning (BRCP) algorithm may be performed on the input weight pattern to generate an output weight pattern for the machine learning model. The block-wise row and column pruning algorithm may include dividing the input weight pattern into a plurality of sub-patterns. For example, each row of the input weight pattern may be divided into a first number of sub-rows (i.e., sub-rows of a first number of sub-patterns), and each column of the input weight pattern may be divided into a second number of sub-columns (i.e., sub-columns of a second number of sub-patterns). The block-wise row and column pruning algorithm may also include pruning sub-columns and sub-rows from a plurality of sub-patterns to achieve a predetermined column sparsity and a predetermined row sparsity, respectively, under the constraint that at least one sub-row in each row of the input weight pattern is not pruned. In some embodiments, the first number may be the same as the second number or may be different from the second number.

[0007] The embodiments of the present disclosure provide a structured pruning method, system, and computer-readable medium, which reduce the storage and computation costs associated with weight patterns. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The detailed description is described in part with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numerals in different drawings indicates similar or identical items.

[0009] Figure 1 An exemplary environment is shown in which a structured pruning system may be used.

[0010] Figure 2A A more detailed exemplary structured pruning system is shown.

[0011] Figure 2B An exemplary neural network processing architecture that can be used to implement a structured pruning system is shown.

[0012] Figure 2C An exemplary cloud system including an exemplary neural network processing architecture to implement an exemplary structured pruning system is shown.

[0013] Figure 3 An exemplary structured pruning method is shown.

[0014] Figure 4 An exemplary block-wise row and column pruning (BRCP) algorithm is shown.

[0015] Figure 5 An exemplary decoding process is shown.

[0016] Figure 6 An exemplary conversion from an output weight pattern obtained by the BRCP algorithm to a compressed weight pattern in a compact form is shown.

[0017] Figure 7A 、 7B 7C and 7C show an exemplary hierarchical hardware design for the BRCP algorithm.

[0018] Figure 8A and 8B shows a data flow that shows Figure 7A 、 7B and 7C illustrate how the exemplary hardware design effectively supports the BRCP algorithm.

[0019] Figure 9 An exemplary row alignment strategy is shown. DETAILED DESCRIPTION

[0020] As mentioned above, existing machine learning models (e.g., neural network models) used for any real-world applications typically include large-size weight matrices, which take up a lot of storage space and require a large number of computational operations (e.g., multiplication and addition / accumulation operations) on the weight matrices to generate predictions or deterministic outputs for real-world applications. This undoubtedly affects the usability and performance of machine learning models in daily life applications and may limit the use of machine learning models in large-scale computing systems with large storage and computing capabilities.

[0021] The present disclosure describes an exemplary structured pruning system. The structured pruning system can convert the input weight pattern (e.g., weight matrix) of a machine learning model into an output weight pattern that is sparser than the input weight pattern in a structured manner, and can further compress the output weight pattern in a hardware-friendly manner, that is, organize the compressed weight pattern in a manner that is convenient for calculation using hardware. In some embodiments, the recognition accuracy of a neural network model using an output weight pattern or a compressed weight pattern can be similar to the recognition accuracy of a neural network model using an input weight pattern. In some embodiments, the machine learning model may include a learning model having one or more weight patterns, which may include, but is not limited to, a neural network model (e.g., a recurrent neural network, a long short-term memory network, etc.), a clustering model (e.g., a hierarchical clustering model, etc.), a Bayesian network, etc.

[0022] In some embodiments, a structured pruning system may receive an input weight pattern of a machine learning model (e.g., a neural network model) and apply a block row and column pruning (BRCP) algorithm to the input weight pattern to generate an output weight pattern. In some embodiments, the BRCP algorithm may include dividing the input weight pattern into a plurality of sub-patterns (e.g., blocks of n rows and m columns, where n and m are positive integers greater than 1), wherein each row of the input weight pattern includes a first number of sub-rows of sub-patterns, and each column of the input weight pattern includes a second number of sub-columns of sub-patterns. The BRCP algorithm may then include pruning sub-columns and sub-rows from the plurality of sub-patterns to achieve a predetermined column sparsity and a predetermined row sparsity, respectively, under the constraint that at least one sub-row in each row of the input weight pattern is not pruned. In some embodiments, pruning a sub-column or sub-row corresponds to setting all elements in the sub-column or sub-row to zero.

[0023] In some embodiments, when obtaining the output weight pattern, the structured pruning system can further compress the output weight pattern by removing the pruned sub-columns and sub-rows from the output weight pattern to generate a compressed weight pattern, which is convenient for parallel calculation by pipeline hardware or multiple processing units such as ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), or other hardware.

[0024] In some embodiments, the functions performed by the structured pruning system described in this disclosure may be performed by multiple separate units or services. For example, a receiving service may receive an input weight pattern, a pruning service may apply a BRCP algorithm to the input weight pattern to generate an output weight pattern, and a compression service may compress the output weight pattern into a hardware-friendly compact or dense weight pattern.

[0025] Furthermore, although in examples described in this disclosure, the structured pruning system may be implemented as software and / or hardware installed in a single device, in other examples, the structured pruning system may be implemented and distributed across multiple devices or as a service provided in one or more servers on a network and / or in a cloud computing architecture.

[0026] This disclosure describes a number of different implementations and embodiments. Below is a description of an example framework suitable for practicing various implementations. Next, this disclosure describes an exemplary system, device, and process for implementing a structured pruning system.

[0027] Exemplary Systems

[0028] Figure 1 An exemplary environment 100 is shown that can be used to implement a structured pruning system. The environment 100 can include a structured pruning system 102. In this example, the structured pruning system 102 is described as existing as a single entity. In some examples, the structured pruning system 102 can be included in a client device 104. Alternatively, the structured pruning system 102 can include one or more client devices. In other examples, the structured pruning system 102 can be included in a plurality of servers 106-1, 106-2, ..., 106-N (collectively referred to as servers 106) that communicate data with each other via a network 108. In some embodiments, the plurality of servers 106 can exist as a cloud or data center.

[0029] In some embodiments, the client device 104 can be implemented as any of a variety of computing devices, including but not limited to desktop computers, notebook or portable computers, handheld devices, netbooks, Internet appliances, tablet or slate computers, mobile devices (e.g., mobile phones, personal digital assistants, smart phones, etc.), etc., or a combination thereof. The client device 104 can communicate data with multiple servers 106 via the network 108 and can request services from the multiple servers 106.

[0030] In some embodiments, the functionality of the structured pruning system 102 can be provided by or included in the client device 104. In some instances, the functionality of the structured pruning system 104 can be included in and / or distributed across multiple servers 106. For example, a first server in the multiple servers 106 can include some functionality of the structured pruning system 102, while other functionality of the structured pruning system 102 can be included in a second server in the multiple servers 106. Additionally or alternatively, the functionality of the structured pruning system 102 can be included in and distributed across the client device 104 and the multiple servers 106. Furthermore, in some embodiments, for example, some or all functionality of the structured pruning system 102 can be included in a cloud computing system or architecture and provided as a service to the client device 104.

[0031] The network 108 may be a wireless or wired network, or a combination thereof. The network 108 may be a collection of individual networks that are interconnected and function as a single large network (e.g., the Internet or an intranet). Examples of these individual networks include, but are not limited to, telephone networks, cable networks, local area networks (LANs), wide area networks (WANs), and metropolitan area networks (MANs). In addition, the individual networks may be wireless or wired networks, or a combination thereof. A wired network may include electrical carrier connections (e.g., communication cables, etc.) and / or optical carriers or connections (e.g., fiber optic connections, etc.). A wireless network may include, for example, a WiFi network, other radio frequency networks (e.g., Zigbee, etc.

[0032] In some embodiments, the structured pruning system 104 may receive instructions from a user of the client device 104 to convert an input weight pattern associated with a machine learning model (e.g., a neural network model) into an output weight pattern that occupies less storage space and requires fewer computational operations (e.g., multiplication and addition / accumulation operations) to produce an output from the input of the neural network model. In response to receiving the input weight pattern, the structured pruning system 104 may perform pruning and compression operations to obtain a sparser and less computationally intensive weight pattern from the input weight pattern, and return the obtained weight pattern to the client device 104, or store the obtained weight pattern in a designated location, such as a storage device of one of the one or more servers 106.

[0033] Exemplary structured pruning system

[0034] Figure 2A2 shows a more detailed structured pruning system 102. In some embodiments, the structured pruning system 102 includes, but is not limited to, one or more processors 202, memory 204, one or more applications or services 206 (e.g., a pruning application or service, a compression application or service, etc.), and program data 208. In some embodiments, the structured pruning system 102 may also include a network interface 210 and an input / output (I / O) interface 212. The processor 202 is configured to execute instructions received from the network interface 210, received from the I / O interface 212, and / or stored in the memory 204. Additionally or alternatively, some or all of the functionality of the structured pruning system 102 may be implemented using an ASIC (Application-Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or other hardware provided in the structured pruning system 102. In some embodiments, one or more processors 202 may include, but are not limited to, a neural network processing unit (NPU), a graphic processing unit (GPU), a tensor processing unit (TPU), a microprocessor, an application-specific instruction-set processor (application-specific instruction-set processor), a physics processing unit (PPU), a digital signal processor, etc.

[0035] Memory 204 may include computer-readable media in the form of volatile memory such as random access memory (RAM) and / or non-volatile memory such as read-only memory (ROM) or flash RAM. Memory 204 is an exemplary computer-readable medium.

[0036] Computer-readable media can include volatile or non-volatile types, removable or non-removable media, which can use any method or technology to implement information storage. Information can include computer-readable instructions, data structures, program modules or other data. Exemplary computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electronically erasable programmable read-only memory (EEPROM), flash memory or other internal storage technology, compact disk read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage device, cassette tape, disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include temporary media, for example, modulated data signals and carrier waves.

[0037] In some embodiments, the structured pruning system 102 may further include a weight pattern database 214 configured to store data of resultant weight patterns obtained after pruning and compression (eg, data of output weight patterns and / or compressed weight patterns).

[0038] In some embodiments, the structured pruning system 102 can be implemented using a neural network processing architecture or a cloud system including a neural network processing architecture to further improve throughput and power efficiency. Additionally or alternatively, the structured pruning system 102 can also be implemented using any accelerator capable of performing parallel computing.

[0039] Figure 2BAn exemplary neural network processing architecture 216 that can be used to implement the structured pruning system 102 is shown. In some embodiments, the neural network processing architecture 216 may include a heterogeneous computation unit (HCU) 218, a host unit 220, and a host memory 222. The heterogeneous computation unit 218 may include specialized computing devices or hardware for facilitating and performing neural network computation tasks. By way of example and not limitation, the heterogeneous computation unit 218 may perform algorithmic operations including operations associated with machine learning algorithms.

[0040] In some embodiments, the heterogeneous computing unit 218 may be an accelerator, which may include, but is not limited to, a neural network processing unit (NPU), a graphics processing unit (GPU), a tensor processing unit (TPU), a microprocessor, a special instruction set processor, a physical processing unit (PPU), a digital signal processor, etc.

[0041] In some embodiments, the heterogeneous computing unit 218 may include one or more computing units 224, a memory hierarchy 226, a controller 228, and an interconnect unit 230. The computing unit 224 can access the memory hierarchy 226 to read and write data in the memory hierarchy 226, and can further perform operations such as arithmetic operations (e.g., multiplication, addition, multiply-accumulate, etc.) on the data. In some embodiments, the computing unit 224 may also include multiple engines that are configured to perform various types of operations. As an example and not limitation, the computing unit 224 may include a scalar engine 232 and a vector engine 234. The scalar engine 232 can perform scalar operations, such as scalar product, convolution, etc. The vector engine 234 can perform vector operations, such as vector addition, vector product, etc.

[0042] In some embodiments, the memory hierarchy 226 may include on-chip memory (e.g., four 8GB second generation high bandwidth memory (HBM2) blocks) used as main memory. The memory hierarchy 226 may be configured to store data and executable instructions and allow, for example, other components of the neural network processing architecture 216 (e.g., heterogeneous compute unit (HCU) 218 and host unit 220), heterogeneous compute unit 218 (e.g., compute unit 224 and interconnect unit 230), and / or devices external to the neural network processing architecture 216 to access the stored data and / or stored instructions at high speed.

[0043] In some embodiments, the interconnect unit 230 can provide or facilitate communication of data and / or instructions between the heterogeneous computing unit 218 and other devices or units external to the heterogeneous computing unit 218 (e.g., the host unit 220, one or more other HCUs). In some embodiments, the interconnect unit 230 can include a peripheral component interconnect express (PCIe) interface 236 and an inter-chip connection 238. The PCIe interface 236 can provide or facilitate communication of data and / or instructions between the heterogeneous computing unit 218 and the host unit 220. The inter-chip connection 238 can be used as an inter-chip bus to connect the heterogeneous computing unit 218 to other devices such as other HCUs, off-chip memory, and / or peripheral devices.

[0044] In some embodiments, the controller 228 may be configured to control and coordinate the operations of other components included in the heterogeneous computing unit 218. For example, the controller 228 may control and coordinate different components (e.g., the scalar engine 232, the vector engine 234, and / or the interconnect unit 230) in the heterogeneous computing unit 218 to facilitate parallel operations or synchronous operations between these components.

[0045] In some embodiments, host memory 222 may be off-chip memory, for example, memory of one or more processing units of a host system or device that includes neural network processing architecture 216. In some embodiments, host memory 222 may include DDR memory (e.g., DDR SDRAM) or the like, and may be configured to store large amounts of data at a slower access speed than on-chip memory integrated within one or more processing units to act as a higher-level cache.

[0046] In some embodiments, the host unit 220 may include one or more processing units (e.g., an X86 central processing unit (CPU)). In some embodiments, a host system or device having the host unit 220 and the host memory 222 may further include a compiler (not shown). A compiler may be a program or computer software configured to convert computer code written in a specific programming language into instructions that can be read and executed by the heterogeneous computing unit 218. In machine learning applications, the compiler may perform various operations, which may include, but are not limited to, preprocessing, lexical analysis, syntax parsing, semantic analysis, converting an input program into an intermediate representation, code optimization and code generation, or any combination thereof.

[0047] Figure 2CAn exemplary cloud system 240 is shown that includes a neural network processing architecture 216 to implement the structured pruning system 102. The cloud system 240 can provide cloud services with machine learning and artificial intelligence (AI) capabilities, and the cloud system 240 can include multiple servers, for example, servers 242-1, 242-2, ... 242-K (collectively referred to as servers 242), where K is a positive integer. In some embodiments, one or more of the servers 242 can include the neural network processing architecture 216. By using the neural network processing architecture 216, the cloud system 240 can provide some or all of the functionality of the structured pruning system 102, as well as other machine learning and artificial intelligence capabilities such as image recognition, facial recognition, translation, 3D modeling, etc.

[0048] In some embodiments, although a cloud system 240 is described above, in some instances, the neural network processing architecture 216 that provides some or all of the functionality of the structured pruning system 102 can be deployed in other types of computing devices, which may include, but are not limited to, mobile devices, tablet computers, wearable devices, desktop computers, etc.

[0049] Exemplary Methods

[0050] Figure 3 is a schematic diagram describing the structured pruning method. Figure 3 The method can but is not required to Figure 1 It is not necessary to use Figure 2A For ease of explanation, refer to Figure 1 and Figure 2A Method 300 is described. However, method 300 may alternatively be implemented in other environments and / or using other systems.

[0051] Method 300 is described in the general context of computer-executable instructions. Typically, computer-executable instructions can include routines, programs, objects, components, data structures, processes, modules, functions, etc. that perform specific functions or implement specific abstract data types. In addition, the exemplary method is described as a set of blocks in a logical flow diagram, which represents a sequence of operations that can be implemented in hardware, software, firmware, or a combination thereof. The order in which the method is described is not intended to be interpreted as a limitation, and any number of the described method blocks can be combined in any order to implement the method or an alternative method. In addition, a single block can be omitted from the method without departing from the spirit and scope of the subject matter described herein. In the context of software, a block represents a computer instruction, and when the computer instruction is executed by one or more processors, the one or more processors perform the operation. In the context of hardware, some or all of the blocks can represent an application specific integrated circuit (ASIC) or other physical component that performs the operation.

[0052] Return Reference Figure 3 , at block 302 , the structured pruning system 102 may receive an input weight pattern.

[0053] In some embodiments, the structured pruning system 102 may receive data of an input weight pattern from the client device 104. The input weight pattern may be a weight pattern obtained after training a machine learning model using a training sample set through a training algorithm. In some embodiments, the machine learning model may include a learning model having a weight pattern, such as a neural network model. Without loss of generality, for the sake of description, a neural network model is used below as an example of a machine learning model. The following description may also be applicable to any other machine learning model that includes a weight pattern, some elements of which may be pruned to achieve the purpose of structured pruning described in this disclosure.

[0054] In some embodiments, the input weight pattern can be a data structure storing weight information of a layer of the neural network model. Alternatively, the input weight pattern can be one or more data structures, each of which stores weight information of a corresponding layer in multiple layers of the neural network model.

[0055] In some embodiments, the data structure storing weight information of a layer of a neural network model may include, for example, a matrix of size N×M, where N and M are positive integers greater than 1, a plurality of arrays having at least one array storing non-zero weight values and another array storing corresponding indices of these non-zero weight values, and the like. In this example, for illustration, the input weight pattern is described as a data structure in a matrix format associated with a layer of the neural network model, and the data structures of other layers of the neural network model can be processed in a similar manner. In addition, in the case where the data structure is not in a matrix format, an appropriate conversion can be applied to convert the data structure into a matrix format.

[0056] At block 304 , the structured pruning system 102 may obtain an output weight pattern from the input weight pattern based on a specified pruning algorithm.

[0057] In some embodiments, the structured pruning system 102 may prune the input weight pattern to obtain the output weight pattern by applying a specified pruning algorithm. In some embodiments, the specified pruning algorithm may include a block row column pruning (BRCP) algorithm.

[0058] By way of example and not limitation, at block 304-2, the BRCP algorithm may include dividing the input weight pattern into a plurality of sub-patterns, wherein each row of the input weight pattern includes a first number of sub-rows of sub-patterns, and each column of the input weight pattern includes a second number of sub-columns of sub-patterns. At block 304-4, the BRCP algorithm may further include pruning sub-columns and sub-rows from the plurality of sub-patterns to achieve a predetermined column sparsity and a predetermined row sparsity, respectively, subject to the constraint that at least one sub-row in each row of the input weight pattern is not pruned. Further details of the BRCP algorithm will be described in later sections.

[0059] In some embodiments, the structured pruning system 102 may divide the input weight pattern into multiple sub-patterns. As an example and not limitation, the structured pruning system 102 may divide the input weight pattern (e.g., a weight matrix of size N×M) into K banks, each of which is divided into L blocks (i.e., sub-patterns), where N, M, K, and L are integers greater than 1, N is greater than L, and M is greater than K. In other words, each row of the input weight pattern is divided into (and includes) K sub-rows, each sub-row coming from a corresponding sub-pattern in the K corresponding sub-patterns. Similarly, each column of the input weight pattern is divided into (and includes) L sub-columns, each sub-column coming from a corresponding sub-pattern in the L corresponding sub-patterns. More details about this division can be found in the following sections.

[0060] In some embodiments, pruning a subcolumn (or subrow) may include setting the elements in the subcolumn (or subrow) to zero. In some embodiments, the norm of a vector (e.g., subcolumn or subrow) may correspond to the square root of the sum of the squares of each element of the vector. In some embodiments, the sparsity of the data (e.g., in matrix form) is a measure of zero values in the data (i.e., a measure of the percentage of elements with zero values in the matrix). The high sparsity of the data corresponds to a value of zero for a large number of elements in the data. For example, the sparsity of data in a certain structure (e.g., matrix, column, or row, etc.) is 90%, meaning that the value of 90% of the elements in the data (i.e., elements in structures such as matrix, column, or row) is zero.

[0061] In some embodiments, pruning sub-columns from the plurality of sub-patterns to achieve a predetermined column sparsity may include pruning sub-columns of the sub-pattern in response to determining that a norm of the sub-columns of the sub-pattern is less than a predetermined sub-column threshold.

[0062] In some embodiments, under the constraint that at least one sub-row in each row of the input weight pattern is not pruned, pruning sub-rows from multiple sub-patterns to achieve a predetermined row sparsity may include pruning sub-rows of the sub-pattern in response to determining that the norm of the sub-rows of the sub-pattern is less than a predetermined sub-row threshold.

[0063] Additionally or optionally, under the constraint that at least one sub-row in each row of the input weight pattern is not pruned, pruning sub-rows from multiple sub-rows to achieve a predetermined row sparsity may include determining that the corresponding norms of all sub-rows in a specific row of the input weight pattern are less than a predetermined sub-row threshold; and retaining at least one sub-row with the highest norm among the corresponding norms of all sub-rows in the specific row of the input weight pattern, and pruning the remaining sub-rows in the specific sub-row of the input weight pattern.

[0064] In some embodiments, the sub-column threshold and the sub-row threshold may be determined based on a predetermined column sparsity and a predetermined row sparsity, respectively. By way of example and not limitation, corresponding sub-column norms and sub-row norms of a plurality of sub-patterns in the input weight pattern may be calculated, and cutoff values of the sub-column norm and the sub-row norm that achieve the predetermined column sparsity and the predetermined row sparsity may be determined and set as the sub-column threshold and the sub-row threshold, respectively.

[0065] In some embodiments, if the structured pruning system 102 accesses a neural network model associated with an input weight pattern, the structured pruning system 102 may also determine the accuracy of the neural network model that adopts the output weight pattern. Specifically, as described above, the input weight pattern may be a weight pattern obtained after training the neural network model using a training sample set through a training algorithm, and the accuracy of the neural network model that adopts the input weight pattern may be determined using a test sample set. In some embodiments, after obtaining an output weight pattern from the input weight pattern using a specified pruning algorithm (e.g., a BRCP algorithm), the structured pruning system 102 may also determine the accuracy of the neural network model that adopts the output weight pattern using the same test sample set, and compare the determined accuracy of the neural network model that adopts the output weight pattern with the accuracy of the neural network model that adopts the input weight pattern.

[0066] If the accuracy of the neural network model with the output weight pattern is not less than a predetermined percentage (e.g., 1%, 2%, 3%, etc.) of the accuracy of the neural network model with the input weight pattern, the structured pruning system 102 may accept the output weight pattern. Otherwise, the structured pruning system 102 may automatically reduce the column sparsity and / or row sparsity and reapply the BRCP algorithm on the input weight pattern to generate a new output weight pattern until the neural network with the new output weight pattern has an accuracy that is not less than the predetermined percentage of the accuracy of the neural network model with the input weight pattern.

[0067] Optionally, the structured pruning system 102 may notify a user of the client device 104 that the accuracy of the neural network model employing the output weight pattern has decreased by more than a predetermined percentage, and await instructions from the user as to whether this decrease in accuracy is acceptable or whether the structured pruning system 102 needs to execute the pruning algorithm again using a different column sparsity and / or row sparsity.

[0068] At block 306 , the structured pruning system 102 may compress the output weight pattern to generate a compressed weight pattern in a compact form.

[0069] In some embodiments, the structured pruning system 102 may compress the output weight pattern by removing sub-columns and sub-rows in which the values of the elements are all zeros from the output weight pattern to produce a compressed weight pattern in a compact form.

[0070] At block 308 , the structured pruning system 102 may store the compressed weight pattern.

[0071] In some embodiments, the structured pruning system 102 may store the compressed weight pattern in the memory 204, eg, the weight pattern database 210, or send the compressed weight pattern to the client device 104 or one of the one or more servers 106 for storage.

[0072] Any action of any method described herein may be performed, at least in part, by a processor or other electronic device based on instructions stored on one or more computer-readable media. By way of example and not limitation, any action of any method described herein may be performed under the control of one or more processors configured with executable instructions that may be stored on one or more computer-readable media.

[0073] Exemplary structured pruning algorithm

[0074] As described in the foregoing description, the structured pruning system 102 can apply a specified pruning algorithm, called a block row column pruning (BRCP) algorithm, to an input weight pattern to generate an output weight pattern. To further understand the BRCP algorithm, the algorithm is described in detail below from three aspects: (1) algorithm prototype, which aims to achieve high overall storage or memory reduction of weights and indices, and achieve efficient decoding; (2) algorithm hardware co-design, which is achieved by analyzing the characteristics of the algorithm and one or more constraints that may be encountered in hardware design; and (3) hardware-aware algorithm design, which uses the characteristics of the algorithm and one or more constraints as guidance.

[0075] In some embodiments, pruning an entire column in the weight pattern (e.g., weight matrix) of a layer of a neural network is equivalent to removing the corresponding input activation. On the other hand, pruning an entire row in the weight pattern is equivalent to removing the corresponding output activation. Specifically, due to the loss of activation information, these two types of pruning can only achieve a limited pruning rate. In contrast, retaining activation information can be beneficial for achieving a high pruning rate.

[0076] In some embodiments, the BRCP algorithm is configured to avoid removing entire columns and / or entire rows, thereby facilitating high pruning rates by avoiding loss of activation information.

[0077] Table 1 shows an example of the BRCP algorithm in detail. Figure 4 An exemplary block row column pruning (BRCP) algorithm 400 is shown. According to Table 1 and Figure 4 , the BRCP algorithm first converts the weight pattern (in this case, the weight matrix W∈R m×n ) is divided into multiple heaps (two heaps in this case, namely W0∈R m×n1 and W1∈R m×n2, where m is the number of rows and n is the number of columns, ). In this example, although the weight pattern is divided into two piles, in other cases, the weight pattern can be divided into more than two piles, for example, three piles, four piles, etc. In some embodiments, the number of piles divided from the weight pattern can depend on the size of the weight pattern, the degree of granularity used to explore sparsity, the number of processing units available for parallel processing of the piles, or any other criteria set by the administrator of the structured pruning system 102.

[0078] Table 1: BRCP algorithm

[0079]

[0080]

[0081] In some embodiments, after partitioning the weight patterns into multiple piles (e.g., two piles in this example), the BRCP algorithm may then partition each pile into blocks (i.e., sub-patterns). where b is the number of rows (i.e., subrows) in each block (i.e., subpattern), In some embodiments, column pruning can be performed individually within each block, rather than within the entire weight pattern (i.e., the entire weight matrix W). In this example, since the weight pattern is divided into two piles, each row of the weight pattern is also divided into two half-rows (i.e., two sub-rows). In some embodiments, however, if the weight pattern is divided into more than two piles, each row of the weight pattern can be divided into more than two sub-rows. The two piles are only for illustration, and the algorithm described below can be extended and applied when the weight pattern is divided into more than two piles.

[0082] After performing column pruning, row pruning can be performed on the entire weight pattern (i.e., the entire weight matrix W), with the (heap-specific) constraint that for any row in the weight pattern, at most one of the two half-rows is allowed to be pruned, or in other words, at least one of the two half-rows is not pruned. This constraint prevents any row in the weight pattern from being completely pruned, thereby avoiding deletion of the corresponding output activations. In some embodiments, if each row of the weight pattern can be divided into more than two sub-rows, the constraint may include that at least one sub-row of any row of the weight pattern is not pruned.

[0083] In some embodiments, in order to make the output weight pattern (i.e., the neural network model that employs the output weight pattern) hardware-friendly to achieve high speed improvements, an algorithm-hardware co-design approach can be used so that any optimization of the pruning algorithm can be hardware-aware. In some embodiments, for a pruned matrix such as the output weight pattern, it is important to know the coordinates of the activations associated with the retained weights (i.e., the elements in the output weight pattern that are retained and not pruned due to a pruning algorithm such as the BRCP algorithm). Therefore, during the computation, decoding is required to select the activations associated with the retained weights from the inputs (i.e., the input vectors) of the neural network layer associated with the output weight pattern. Figure 5 An exemplary decoding process 500 is shown. Figure 5 In

[15] , each retained weight can have an index for decoding. If irregular or random pruning is performed, these indices may be arbitrary and random, requiring a search over the entire input vector. Given that the bandwidth for decoding may be limited, decoding cannot generally be accelerated for irregular or random pruning. Therefore, increasing the number of weights (or elements in a weight pattern) shared by a common index can help improve or speed up the performance of computations associated with the weight pattern.

[0084] In some embodiments, the b value (the number of sub-rows in a sub-pattern) in the BRCP algorithm is a factor that determines the number of weights (i.e., elements in a weight pattern) shared by a common index. Therefore, the larger the b value, the fewer the number of indices, and the faster the decoding can be. In some embodiments, given a block (i.e., sub-pattern) with a sparsity of p, the time taken to decode half a row of the weight pattern in this example is This is also the time it takes to decode the entire block. Therefore, the average delay in decoding a block is In some embodiments, if sufficient hardware resources are available, multiple blocks of the two heaps can be decoded in parallel to improve overall throughput. If g represents the number of groups decoded in parallel, the average time spent decoding each row of the weight pattern can be expressed as follows:

[0085]

[0086] In some embodiments, to improve the utilization of hardware resources, the decoder and the processing engine (PE) can operate at the same speed without waiting for each other. For example, if s represents the number of available processing engines, then the time taken to calculate a group satisfies the constraints defined in the following formulas (2) and (3). Formula (2) is:

[0087]

[0088] Formula (2) can be simplified as:

[0089] s=b (3)

[0090] Based on the above analysis, the BRCP algorithm can improve both the pruning rate and pruning performance. In some embodiments, for hardware design, s can be set equal to b to ensure the utilization of the decoder and processing engine. In addition, in some embodiments, for example, as long as hardware resources are available or the cost of hardware resources is acceptable, g can be set to be large enough to improve the parallelism of decoding and computing.

[0091] In some embodiments, for algorithm design, formula (2) indicates that increasing Can help improve decoding performance. Therefore, b can be based on In some embodiments, the overall sparsity p is determined by the optimal value of , which may also be subject to the on-chip memory budget. c and row sparsity p r , we can also calculate the column sparsity p c Multiple constraints are imposed to allow for hardware friendly design. By way of example and not limitation, the multiple constraints may include: (1) all blocks (ie, sub-patterns) have the same column sparsity p c , to reduce the complexity of decoding; (2)n·p c is a multiple of s so that the processing engine can run at full speed. This can be achieved by rounding off or changing the row sparsity p accordingly. r to achieve.

[0092] In some embodiments, block column pruning and heap-specific row pruning enable the BRCP algorithm to achieve a higher pruning rate for the output weight pattern without loss of accuracy compared to the input weight pattern, while significantly reducing the complexity of indexing and decoding. In addition, the BRCP algorithm performs restoration and calculation of the weight pattern or matrix (i.e., the output weight pattern or matrix) without converting the weight pattern to the frequency domain, so the reduction in storage or memory caused by the BRCP algorithm can lead to an overall reduction in calculation. In addition, by constraining each block (i.e., each sub-pattern) to have the same column sparsity p c , and divide the blocks (i.e., sub-patterns) in the row direction, the BRCP algorithm is hardware-friendly and can achieve excellent acceleration. In some embodiments, the output weight pattern as a sparse weight pattern can be compressed into a compact form using the relative positions of the retained elements (i.e., weights). For example, Figure 6 An exemplary conversion or compression 600 from an output weight mode to a compressed weight mode is shown.

[0093] Exemplary Hardware Architecture Design

[0094] Figure 7A 、 7B 7C and 7C show an exemplary hierarchical hardware design supporting the BRCP algorithm. Figure 7A An exemplary hardware design of a Long Short Term Memory (LSTM) layer is shown. Figure 7B An exemplary hardware design of a gate is shown. Figure 7C The following figure shows an example hardware design of a Sparse Group Compute Engine (SGCE). In this example, although long short-term memory (LSTM) is used for illustration, the described BRCP algorithm and hardware design can also be applied to other types of neural network models. The top-level blocking block is the gate of the LSTM layer. As an example and not a limitation, Figure 4 As shown, the gate may include multiple functional gates, such as a cell gate, a forget gate, an input gate, and an output gate. In some embodiments, the functional gate may include two dot product engines (DPEs) and an activation function unit, and is responsible for performing the calculation f(xW x +hW h + bias), where f(.) represents a function, for example, the function can be a sigmoid or tanh function.

[0095] In some embodiments, multiple weight blocks (i.e., weight sub-patterns) can be processed in parallel to improve computational throughput. In this example, each dot product engine (DPE) can be constructed with multiple sparse group computation engines (SGCEs) that are used to process a group of blocks, such as Figure 7B As shown, one block from heap 0 and another block from heap 1. In this example, although two blocks in a group are described, a group can include more than two blocks, and the Sparse Group Compute Engine (SGCE) can be used to process more than two blocks in a group accordingly.

[0096] In some embodiments, the SGCE may include multiple Sparse Block Compute Engines (SBCEs), which may serve as basic processing primitives. For example, and not limitation, Figure 7CAs shown, the SBCE can take as input the decoded activations and corresponding weights and compute the partial sum (PS) of the block. In addition, each SBCE can use an activation buffer for temporarily storing the decoded activations. In some embodiments, the decoded activations and corresponding weights can be input to the processing engine to compute multiplications and accumulations (i.e., additions).

[0097] Due to the nature of row pruning, subrows from different heaps may not align with each other. In some embodiments, to address this potential misalignment, row alignment logic can be used to skip calculations for pruned subrows and ensure that partial sums of subrows of the same row can be correctly added or accumulated.

[0098] In some embodiments, since all SGCEs follow the same processing pipeline, multiple functional components can be shared between SGCEs. These functional components may include, but are not limited to, weight memory, index memory, control logic for weight and index loading / storage, index decoding logic, and processing engines. On the other hand, each SGCE may have its own input and hidden state. In some embodiments, a piecewise linear function can be implemented using a lookup table to approximate the sigmoid and tanh functions used to implement the activation function. In addition, a divide-and-conquer strategy can be used to design the control logic. In some embodiments, a microcontroller can be deployed in each DPE to maximize the use of shared resources and ensure functional correctness. Optionally, in some embodiments, global control logic can exist across multiple DPEs or multiple functional gates. In some embodiments, dividing complex global control into multiple microcontrollers can significantly reduce the complexity of the design, and because each DPE can operate independently, it can have outstanding scalability, while local controllers can help reduce communication delays.

[0099] Figure 8A and 8B shows a data flow that shows Figure 7A 、 7B and 7C to effectively support the BRCP algorithm. As described above, the input weight pattern (and therefore the output weight pattern) can be divided into multiple piles. For the sake of description, without loss of generality, two piles are described in this example. However, the following description can be generalized to the case when the weight pattern is divided into more than two piles.

[0100] By way of example and not limitation, Figure 8AAs shown, the output weight pattern (which is a sparse weight matrix in this example) can be compressed and stored in a compact form. The compressed or dense weight matrix can then be divided into two piles (i.e., pile 0 and pile 1 in this example), and each pile can also be divided into multiple blocks. For example, Figure 8B As shown, two blocks (ie, sub-patterns) from different heaps may form a group and may be processed independently by the processing engine.

[0101] In some embodiments, each sub-row in a block can also be divided into vectors. Figure 8B As shown, vectors in the same sub-row are shaded in the same way. In some embodiments, multiple groups can be processed in parallel. Figure 8B As shown, the vectors of each pile are fed into the corresponding processing engine for dot product calculation to produce the partial sum (PS). In some embodiments, the calculation of the pruned sub-row can be skipped, and the row alignment logic can be used to align the corresponding partial sums from the two piles. The final result can be generated and obtained by adding a bias and applying an activation function.

[0102] In some embodiments, the compressed sparse row (CSR) format can also be used to encode the output weight pattern (i.e., sparse weight matrix) to further reduce storage overhead based on the reduction in index storage cost caused by the BRCP algorithm. The CSR format represents a matrix through three (one-dimensional) arrays, which respectively include non-zero values, row extents, and column indices. The CSR format indicates the relative distance between two non-zero elements. In some embodiments, the index can be loaded from the index memory and the index can be sent to the activation detector to generate appropriate activations fed into the SGCE for calculation. An index decoder may be required to select useful activations (e.g., non-zero activations) from the vector. In some embodiments, for performance reasons, the decoder can be kept at high bandwidth, which requires multiple instances.

[0103] In some embodiments, to address the design challenges of balancing and routing these multiple high-bandwidth components (e.g., index decoders), the interface width of the activation detector can be reduced. Figure 7BThe input interface of the activation detector shown can be n bits wide, where n<<m. In some embodiments, the ping-pong buffer can be further used to hide the latency associated with data loading due to the time it takes to calculate the entire weight matrix (i.e., weight pattern). In some embodiments, if one buffer is used for calculation, the other buffer can be used to load data for processing by the next layer. In addition, since the decoding rate is one activation per cycle per decoder, the aggregated output bandwidth can increase linearly with the total number of decoders operating in parallel.

[0104] In some embodiments, it is necessary to add the partial sums of a group of multiple blocks (two blocks in this example) to produce the final result of the dot product. Since row pruning may cause some misalignment between multiple blocks, a row alignment scheme is proposed to solve these misalignments. Figure 9 An exemplary row alignment strategy 900 is shown. In some embodiments, multiple buffers may be used to temporarily store partial sums generated and obtained from a group of multiple blocks, one buffer for each block. Figure 9 As shown, two buffers are used to temporarily store the partial sums generated and obtained by a group of two blocks (for example, the left block from pile 0 and the right block from pile 1). If a half row is pruned, the partial sum can be stored in the buffer by skipping a step, and the buffer can be reset to zero during the skipping step. The summation can be performed by sequentially obtaining the corresponding partial sums from the two buffers. For example, Figure 9 As shown, <l0,l1,…,l i > can represent the non-zero partial sum of the left block, <r0,r1,…,r i > can represent the non-zero partial sum of the right block, where i represents the row index in the corresponding block. The final dot product result can be calculated by d0=l0+r0, d1=l1+0, d2=0+r2, etc. After obtaining the partial sum, the corresponding buffer entry is released to accept a new partial sum.

[0105] In some embodiments, due to the hierarchical and modular structural design, BRCP can achieve different degrees of scalability and pruning rate with high flexibility. Specifically, the hardware-aware pruning algorithm design described in the above description can facilitate the utilization of hardware resources. For example, by adjusting the column sparsity p c, the number of retained weights can be set to a multiple of the number of processing engines. Therefore, the control design can be simplified because there is no need to zero-fill the sub-columns. In addition, the number of processing engines can be designed to match the number of blocks (i.e., sub-patterns) in the group, so the decoding process can have the same throughput as the processing engines, thereby fully utilizing all units. In addition, since each DPE can work independently and the SGPEs in each DPE are isomorphic, the structure can be expanded by the number of gates working in parallel or the number of groups. Since the bandwidth of each memory bank needs to meet the needs of the group, the number of memory banks can also be scaled simultaneously.

[0106] Although the above examples use the division of weight patterns into two piles to illustrate various implementations of the BRCP algorithm and related hardware designs, in some embodiments, the weight patterns can be divided into multiple piles, which can be more than two piles, and the above implementations can be generalized accordingly to apply to this case.

[0107] In some embodiments, the algorithms described above (e.g., BRCP algorithm, decoding process, etc.) are suitable for implementation via various types of hardware platforms, including, but not limited to, central processing units (CPUs), graphics processing units (GPUs), field-programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). The block sizes and pruning rates described above can be configured based on the respective characteristics of different platforms to achieve optimal or desired acceleration performance. For example, due to the flexibility of FPGAs and ASICs used to support the aforementioned decoding process (e.g., selecting corresponding activations for retained weights), the algorithms described in the foregoing description are suitable for implementation via dedicated processing architectures, such as the neural network processing architecture 216 described above, as well as any other FPGA / ASIC-based neural network accelerators.

[0108] In some embodiments, the BRCP algorithm helps reduce the number of parameters of the machine learning model (e.g., the number of non-zero weights of the neural network model) and the number of calculations using the parameters (e.g., weights), thereby accelerating or speeding up the calculations. In addition, if the aforementioned algorithm is implemented using a dedicated processing architecture such as the neural network processing architecture 216 described above and any other FPGA / ASIC-based neural network accelerator, the number of accesses between local memory and external memory can be reduced, and the total number of calculations can also be reduced. In addition, through appropriate configuration, the processing performance of the accelerator can be further improved.

[0109] In some embodiments, for a given precision, a larger block size (i.e., a larger sub-mode size) can be advantageous for achieving higher decoding efficiency because a higher number of weights that can share the same index results in a higher average processing speed for arithmetic calculations. On the other hand, a larger block size can result in a lower achievable sparsity and, therefore, a smaller reduction in the number of achievable calculations. In some embodiments, given a particular block size, a higher precision requirement can result in a lower sparsity, and a lower precision requirement can result in a higher sparsity. Thus, given a particular application and precision requirement, block size and sparsity can be considered together to achieve an optimal or desired computational speedup.

[0110] in conclusion

[0111] Although the embodiments have been described using language specific to structural features and / or methodological acts, it should be understood that the claims are not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as exemplary forms of implementing the claimed subject matter. Additionally or alternatively, some or all operations may be implemented by one or more ASICs, FPGAs, or other hardware.

Claims

1. A structured pruning method for a machine learning model, the method being implemented via layered hardware, the method comprising: receiving an input weight pattern of a neural network model from a computing device and storing the input weight pattern in a weight memory of the layered hardware; An output weight pattern is obtained from the input weight pattern based on a block row and column pruning method, the block row and column pruning method comprising: Dividing the input weight pattern into a mesh of sub-patterns, each row of the input weight pattern comprising a first number of sub-rows of sub-patterns, and each column of the input weight pattern comprising a second number of sub-columns of sub-patterns; sending each of the subpatterns from a weight memory to a corresponding sparse group compute engine of a plurality of sparse group compute engines of the hierarchical hardware, the weight memory being shared and accessed by the plurality of sparse group compute engines; pruning, by a corresponding sparse group computation engine, corresponding one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and Under the constraint that, for each row of the input weight pattern, at least one sub-row of a sub-pattern in the corresponding row of the input weight pattern is not pruned, pruning corresponding one or more sub-rows from each sub-pattern by a corresponding sparse group calculation engine to achieve a predetermined row sparsity; Send the output weight pattern to the computation device.

2. The structured pruning method according to claim 1, wherein: Pruning the corresponding one or more sub-columns includes: Set all weight values in the corresponding subcolumn or subcolumns to zero.

3. The structured pruning method according to claim 1, wherein: The method further comprises: The output weight pattern is compressed in a compact form by removing sub-columns and sub-rows having all-zero weight values from the output weight pattern.

4. The structured pruning method according to claim 1, wherein: Pruning corresponding one or more sub-columns from each sub-pattern of the mesh network of sub-patterns to achieve a predetermined column sparsity comprises: In response to determining that the norm of the sub-column of the sub-pattern is less than a first predetermined threshold, the sub-column of the sub-pattern is pruned.

5. The structured pruning method according to claim 4, wherein: The norm of the sub-column comprises the square root of the sum of the squares of each element of the sub-column.

6. The structured pruning method according to claim 1, wherein: Pruning corresponding one or more sub-rows from each sub-pattern of the mesh network of sub-patterns under the constraint to achieve a predetermined row sparsity includes: In response to determining that the norm of the sub-row of the sub-pattern is less than a second predetermined threshold, the sub-row of the sub-pattern is pruned.

7. The structured pruning method according to claim 6, wherein: Pruning corresponding one or more sub-rows from each sub-pattern of the mesh network of sub-patterns under the constraint to achieve a predetermined row sparsity further comprises: Determining that corresponding norms of all sub-rows in a specific row of the input weight pattern are less than the second predetermined threshold; and At least one sub-row with the highest norm among corresponding norms of all sub-rows in the specific row of the input weight pattern is retained, and the remaining sub-rows in the specific sub-row of the input weight pattern are pruned.

8. The structured pruning method according to claim 1, wherein: Pruning the corresponding one or more sub-columns from each sub-schema to achieve a predetermined column sparsity includes: One or more corresponding sub-columns are pruned from each sub-pattern so that the sparsity of the sub-columns in the output weight pattern is within a predetermined range around a predetermined column sparsity.

9. A computer-readable medium storing executable instructions that, when executed by one or more processors of layered hardware, cause the one or more processors to: receiving an input weight pattern of a neural network model from a computing device and storing the input weight pattern in a weight memory of the layered hardware; An output weight pattern is obtained from the input weight pattern based on a block row and column pruning method, the block row and column pruning method comprising: Dividing the input weight pattern into a mesh of sub-patterns, each row of the input weight pattern comprising a first number of sub-rows of sub-patterns, and each column of the input weight pattern comprising a second number of sub-columns of sub-patterns; sending each of the subpatterns from a weight memory to a corresponding sparse group compute engine among a plurality of sparse group compute engines of the hierarchical hardware, the weight memory being shared and accessed by the plurality of sparse group compute engines; pruning, by a corresponding sparse group computation engine, corresponding one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and Under the constraint that, for each row of the input weight pattern, at least one sub-row of a sub-pattern in the corresponding row of the input weight pattern is not pruned, pruning corresponding one or more sub-rows from each sub-pattern by a corresponding sparse group calculation engine to achieve a predetermined row sparsity; Send the output weight pattern to the computation device.

10. The computer-readable medium of claim 9, wherein: Pruning the corresponding one or more sub-columns includes: Set all weight values in the corresponding subcolumn or subcolumns to zero.

11. The computer-readable medium of claim 9, wherein: The actions also include: The output weight pattern is compressed in a compact form by removing sub-columns and sub-rows having all-zero weight values from the output weight pattern.

12. The computer-readable medium of claim 9, wherein: Pruning corresponding one or more sub-columns from each sub-pattern of the mesh network of sub-patterns to achieve a predetermined column sparsity includes: In response to determining that the norm of the sub-column of the sub-pattern is less than a first predetermined threshold, the sub-column of the sub-pattern is pruned.

13. The computer-readable medium of claim 12, wherein: The norm of the sub-column comprises the square root of the sum of the squares of each element of the sub-column.

14. The computer-readable medium of claim 9, wherein: Pruning corresponding one or more sub-rows from each sub-pattern of the mesh network of sub-patterns under the constraint to achieve a predetermined row sparsity includes: In response to determining that the norm of the sub-row of the sub-pattern is less than a second predetermined threshold, the sub-row of the sub-pattern is pruned.

15. The computer-readable medium of claim 14, wherein: Pruning corresponding one or more sub-rows from each sub-pattern of the mesh network of sub-patterns under the constraint to achieve a predetermined row sparsity further comprises: Determining that corresponding norms of all sub-rows in a specific row of the input weight pattern are less than the second predetermined threshold; and At least one sub-row with the highest norm among corresponding norms of all sub-rows in the specific row of the input weight pattern is retained, and the remaining sub-rows in the specific sub-row of the input weight pattern are pruned.

16. The computer-readable medium of claim 9, wherein: Pruning the corresponding one or more sub-columns from each sub-schema to achieve a predetermined column sparsity includes: One or more corresponding sub-columns are pruned from each sub-pattern so that the sparsity of the sub-columns in the output weight pattern is within a predetermined range around a predetermined column sparsity.

17. A structured pruning system for a machine learning model, the system being implemented using layered hardware, the system comprising: one or more neural network processing units; A memory for storing executable instructions, which, when executed by the one or more neural network processing units, causes the one or more neural network processing units to perform the following actions: receiving an input weight pattern of a neural network model from a computing device and storing the input weight pattern in a weight memory of the layered hardware; An output weight pattern is obtained from the input weight pattern based on a block row and column pruning method, the block row and column pruning method comprising: Dividing the input weight pattern into a mesh of sub-patterns, each row of the input weight pattern comprising a first number of sub-rows of sub-patterns, and each column of the input weight pattern comprising a second number of sub-columns of sub-patterns; sending each of the subpatterns from a weight memory to a corresponding sparse group compute engine among a plurality of sparse group compute engines of the hierarchical hardware, the weight memory being shared and accessed by the plurality of sparse group compute engines; pruning, by a corresponding sparse group computation engine, corresponding one or more sub-columns from each sub-pattern to achieve a predetermined column sparsity; and Under the constraint that, for each row of the input weight pattern, at least one sub-row of a sub-pattern in the corresponding row of the input weight pattern is not pruned, pruning corresponding one or more sub-rows from each sub-pattern by a corresponding sparse group calculation engine to achieve a predetermined row sparsity; Send the output weight pattern to the computation device.

18. The structured pruning system according to claim 17, wherein: Pruning the corresponding one or more sub-columns includes: Set all weight values in the corresponding subcolumn or subcolumns to zero, The actions also include: The output weight pattern is compressed in a compact form by removing sub-columns and sub-rows having all-zero weight values from the output weight pattern.

19. The structured pruning system of claim 17, wherein: Pruning corresponding one or more sub-columns from each sub-pattern of the mesh network of sub-patterns to achieve a predetermined column sparsity includes: In response to determining that the norm of the sub-column of the sub-pattern is less than a first predetermined threshold, the sub-column of the sub-pattern is pruned.

20. The structured pruning system of claim 17, wherein: Pruning corresponding one or more sub-rows from each sub-pattern of the mesh network of sub-patterns under the constraint to achieve a predetermined row sparsity includes: In response to determining that the norm of the sub-row of the sub-pattern is less than a second predetermined threshold, pruning the sub-row of the sub-pattern; Determining that corresponding norms of all sub-rows in a specific row of the input weight pattern are less than the second predetermined threshold; and At least one sub-row with the highest norm among corresponding norms of all sub-rows in the specific row of the input weight pattern is retained, and the remaining sub-rows in the specific sub-row of the input weight pattern are pruned.

Images