Method and device for executing sparse calculation based on artificial intelligence based on mixed mode and dynamic coding

Abstract

Methods, apparatus, systems, and articles of manufacture are disclosed for performing artificial intelligence-based sparse calculations based on mixed mode and dynamic coding. An example apparatus includes a memory, computer readable instructions, and processor circuitry to execute the computer readable instructions to: determine a hybrid sparse pattern of a selected layer of an artificial intelligence (AI)-based model, the hybrid sparse pattern having a sparseness ratio and a block pattern for the selected layer; in response to the sparseness ratio being above a threshold, reducing the sparseness ratio of the selected layer; and in response to the sparseness ratio being below the threshold, adjusting a block mode of the selected layer, the block mode of the selected layer corresponding to the accuracy ratio.

Description

Technical Field

[0001] The present disclosure relates generally to machine learning, and more particularly, to methods and apparatus for performing artificial intelligence-based sparse computations based on hybrid modes and dynamic encoding. Background Art

[0002] Artificial intelligence (e.g., machine learning, deep learning, etc.) has become increasingly popular in recent years. Artificial intelligence can be implemented using neural networks. A neural network is a computing system inspired by the neural network of the human brain. A neural network can receive input and generate output. A neural network includes a plurality of neurons corresponding to weights, which can be trained (e.g., learned, weighted, etc.) based on feedback so that the output corresponds to the desired result. Once the weights are trained, the neural network can make decisions based on any input to generate an output. Neural networks are used in the emerging fields of artificial intelligence and / or machine learning. A deep neural network is a specific type of neural network that includes multiple layers of neurons between the input and the output. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 is a schematic diagram of an example artificial intelligence-based model.

[0004] Figure 2 yes Figure 1 Block diagram of an example implementation of a model training circuit module.

[0005] Figure 3-Figure 5 The diagram shows that the representation can be executed to implement Figure 1 and / or Figure 2 A flowchart of example machine readable instructions for an example model training circuit module.

[0006] Figure 6 An example of dynamic sparse coding is shown.

[0007] Figure 7 is constructed to execute Figure 3-Figure 5 Instructions to achieve Figure 1 and / or Figure 2 Block diagram of an example processing platform for an example model training circuit module.

[0008] Figure 8 yes Figure 7 A block diagram of an example implementation of a processor circuit module.

[0009] Fig. 9 yes Figure 7 A block diagram of another example implementation of a processor circuit module.

[0010] Fig.10 is a block diagram of an example software distribution platform for distributing software (e.g., Figure 3-Figure 5 The invention also provides software corresponding to example computer-readable instructions of the present invention (e.g., software corresponding to example computer-readable instructions of the present invention) for distribution to client devices, such as consumers (e.g., for licensing, sale, and / or use), retailers (e.g., for sale, resale, licensing, and / or sublicensing), and / or original equipment manufacturers (OEMs) (e.g., for inclusion in products to be distributed to, for example, retailers and / or direct purchasing customers).

[0011] The drawings are not drawn to scale. Generally, the same reference numerals will be used throughout the drawings and the attached written description to refer to the same or similar parts. Connection references (e.g., attachment, coupling, connection and engagement) should be interpreted broadly, and unless otherwise stated, may include intermediate members between sets of elements and relative movement between elements. Therefore, connection references do not necessarily assert that two elements are directly connected and are in a fixed relationship to each other. Although the drawings show layers and regions with clean lines and boundaries, some or all of these lines and / or boundaries may be idealized. In fact, boundaries and / or lines may be unobservable, mixed and / or irregular.

[0012] When identifying multiple elements or components that can be mentioned separately, the descriptors "first", "second", "third", etc. are used herein. Unless otherwise specified or understood based on the context of their use, such descriptors are not intended to confer any meaning on the arrangement, physical order or priority, or time ordering in the list, but can be used as a mark for referring to multiple elements or components separately to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" can be used to refer to an element in the detailed description, while the same element can be referred to by different descriptors in the claims, such as "second" or "third". In this case, it should be understood that such descriptors are only used to facilitate reference to multiple elements or components. DETAILED DESCRIPTION

[0013] Artificial intelligence (AI)-based models (such as machine learning models, deep learning models, neural networks, deep neural networks, etc.) are used to perform tasks (e.g., classify data). Ground truth data (e.g., data correctly labeled with a specific classification) can be used to train AI-based models. Training traditional AI models adjusts the weights of neurons in a neural network. After training, the data is input into the trained neural network, and the weights of the neurons are applied (e.g., multiplication and accumulation (MAC)) to the input data to be able to process the input data to perform a function (e.g., classify the data). For example, each neuron can be implemented by a MAC processing element (PE), which obtains the input data and / or output data (e.g., activation data) of the previous layer, and multiplies the input / activation data with the weights developed from the training to generate an output value for the neuron. As used herein, the terms data element and activation are interchangeable and mean the same thing. In particular, as defined herein, a data element or activation is a data partition in a data structure. The output value can be transmitted to a subsequent layer and / or another component (e.g., a classifier for classifying the output data). As used herein, an activation tensor (e.g., also referred to as an activation matrix) is input data and / or output data from a particular layer of an AI-based model, and a weight tensor (e.g., also referred to as a weight matrix) is a set of weights for a particular layer.

[0014] The technique for improving performance and reducing energy consumption is by exploiting the properties of sparsity that are present in large quantities in the network. Sparsity refers to the presence of zeros in weights and activations in AI-based models. Zero-valued activations in AI-based models stem from the processing of layers by activation functions, while zero-valued weights usually appear due to filter pruning or due to the quantization process in AI-based models. These zero-valued activations and weights do not contribute to the results during MAC operations in convolutional and fully connected layers, so they can be skipped during computation and storage. Therefore, machine learning accelerators can exploit this sparsity available in activations and weights to achieve significant acceleration during computation, which leads to power savings because less energy can be used to complete the same work, as well as reducing storage requirements for weights (and activations) via effective compression schemes. Reducing the total amount of data transfer across memory hierarchies and reducing total computation time are critical to improving energy efficiency in machine learning accelerators.

[0015] The examples disclosed herein relate to AI-based models in which fewer than a threshold number of parameters are non-zero (e.g., sparse activation and / or weight data). Sparsification is an optimization technique for reducing computation and memory usage for AI-based workloads. Model sparsity and / or compression should balance the accuracy of the AI-based model with the performance of the AI-based model, because improving the performance of the AI-based model may result in a reduction in the accuracy of the AI-based model. As used herein, structural sparsity (also known as block-wise sparsity) refers to processing matrices that have been divided into non-overlapping blocks, where blocks filled with zeros are considered zero blocks, and blocks that include at least one non-zero element are considered non-zero blocks.

[0016] The examples disclosed herein generate a hybrid sparse mode (also called a hybrid block mode) that balances accuracy and performance based on adjusting the sparsity ratio and block size when generating the hybrid sparse mode. The hybrid sparse mode of a layer of an AI-based model includes the sparsity ratio and block mode of the layer. The sparsity ratio (sparsity ratio) (also called sparse ratio) is the ratio of zero blocks to all blocks in a group. The higher the sparsity, the more non-zero elements there are, which corresponds to a higher accuracy drop but a performance gain. The larger the block size, the higher the accuracy drop, but the greater the performance gain. The hybrid sparse mode is a mixture of one or more sparse modes (also called block modes) with configurable parameters. For example, if block-by-block sparsity is applied to a deep learning model, the size of the block may be different for each general matrix-to-matrix multiplication (GEMM) layer due to accuracy constraints. The sparsity of the input / output layer closer to the model with a smaller block size provides higher accuracy, while the intermediate layers benefit from a larger block size to benefit performance. However, it is difficult or impossible to design a sparsity pattern for each convolution or GEMM layer because the relationship between model weights and accuracy is currently a black box. The examples disclosed herein provide a technique for generating a sparse model corresponding to a hybrid sparsity pattern during training to increase the sparsity ratio to improve performance while maintaining high accuracy.

[0017] In addition, the examples disclosed herein perform a dynamic sparse coding designed for the disclosed hybrid sparse mode to improve hardware efficiency. Non-zero elements are randomly distributed in a sparse matrix, and the hybrid mode may introduce additional randomness. Randomness limits the utilization of modern central processing units (CPUs) and / or modern accelerator single instruction multiple data (SIMD), thereby reducing hardware efficiency. The examples disclosed herein provide a dynamic sparse coding technology instead of traditional static sparse coding, and a corresponding automatically generated core to achieve a balance between performance and accuracy. Dynamic sparse coding is designed for hybrid sparse modes, and coding parameters will be adjusted to increase the density of calculations to obtain better hardware efficiency. In addition, the examples disclosed herein provide a kernel generator for generating a specific kernel corresponding to a specific weight based on a dynamic sparse coding result.

[0018] Typically, implementing a machine learning (ML) / artificial intelligence (AI) system involves two phases, a learning / training phase and an inference phase. In the learning / training phase, a training algorithm is used to train the model to operate according to patterns and / or associations based on, for example, training data. Typically, the model includes internal parameters that guide how input data is converted into output data, such as by converting input data into output data through a series of nodes and connections within the model. In addition, hyperparameters can be used as part of the training process to control how learning is performed (e.g., learning rate, number of layers to be used in a machine learning model, etc.). Hyperparameters are defined as training parameters that are determined before initiating the training process.

[0019] Different types of training can be performed based on the expected output and / or type of ML / AI model. As used herein, labeling refers to the expected output of a machine learning model (e.g., classification, expected output value, etc.). Alternatively, unsupervised training (e.g., for deep learning, subsets of machine learning, etc.) involves inferring patterns from inputs to select parameters for an ML / AI model (e.g., without the benefit of an expected (e.g., labeled) output).

[0020] In the examples disclosed herein, training is performed until a threshold number of actions have been predicted. In the examples disclosed herein, training is performed locally (e.g., in a device) or remotely (e.g., in the cloud and / or at a server). Training can be performed using hyperparameters that control how learning is performed (e.g., learning rate, number of layers to be used in a machine learning model, etc.). In some examples, retraining can be performed. This retraining can be performed in response to a new program being implemented or a new user using the device. Training is performed using training data. When supervised training can be used, the training data is labeled. In some examples, the training data is preprocessed.

[0021] Once training is complete, the model is deployed to serve as an executable construct that processes inputs and provides outputs based on the network of nodes and connections defined in the model. The model is stored locally in memory (e.g., cached and moved to memory after training) or can be stored in the cloud. The model can then be executed by a processor.

[0022] Once trained, the deployed model can operate in the reasoning phase to process data. In the reasoning phase, the data to be analyzed (e.g., real-time data) is input to the model, and the model executes to create output. This reasoning phase can be thought of as the AI ​​"thinking" to generate output based on what was learned from training (e.g., by executing the model to apply the learned patterns and / or associations to real-time data). In some examples, the input data undergoes preprocessing before being used as input for the machine learning model. In addition, in some examples, the output data can undergo post-processing after it is generated by the AI ​​model to convert the output into a useful result (e.g., a display of data, instructions to be executed by the machine, etc.).

[0023] In some examples, the output of the deployed model can be captured and provided as feedback. By analyzing the feedback, the accuracy of the deployed model can be determined. If the feedback indicates that the accuracy of the deployed model is less than a threshold or other criteria, the feedback and updated training data sets, hyperparameters, etc. can be used to trigger training of the updated model to generate an updated deployed model.

[0024] Figure 1 is a schematic diagram of an example model training circuit module 102 for training an example AI-based model 104. The example AI-based model 104 includes an example system memory 106 and example neuron (referred to herein as neuron, computational node, processing element, etc.) layers 110a-c. Figure 1 The neurons 110a-c shown in FIG. 1 include six neurons in three layers, but any number of neurons may be present in any type of configuration. Although described in conjunction with the AI-based model 104 Figure 1 , but the examples disclosed herein can be used in any AI-based system or model that includes weights.

[0025] Figure 1 The example model training circuit module 102 trains the AI-based model 104 by selecting weights (e.g., formed in a vector or matrix) for each of the neurons 110a-c. Initially, the AI-based model 104 is untrained (e.g., the neurons have not yet been weighted and / or are only initially weighted). To train the AI-based model 104, Figure 1The example model training circuit module 102 uses training data (e.g., input data with known classifications and / or output labels) to configure the AI-based model 104 to be able to predict output classifications for input data with unknown classifications. In some examples, the model training circuit module 102 can train the model using a first set of training data and test the model using a second set of training data. If the accuracy of the model is below a threshold based on the results of the test, the model training circuit module 102 can use additional sets of training data to tune (e.g., adjust, further train, etc.) the parameters of the model and continue testing until the accuracy is above the threshold. After the model training circuit module 102 has trained the AI-based model 104, the example model training circuit module 102 stores the corresponding means and deviations for the corresponding neurons 110a-c in the example system memory 106 of the example AI-based model 104. The example model training circuit module 102 can be implemented in the same device as the AI-based model 104 and / or in a separate device (e.g., computing device 114) that communicates with the example AI-based model 104. For example, the model training circuit module 102 can be remotely located, develop weight data locally, and deploy the weight data (e.g., a vector / matrix of weights to be implemented by corresponding neurons 110a-c) to the AI-based model 104 for implementation via the example network 112 (e.g., applying the weights to activations via a MAC operation).

[0026] In some examples, Figure 1 The model training circuit module 102 trains the AI ​​model 102 for sparsity. As described above, the higher the sparsity ratio of the trained weight tensor (e.g., more zero blocks and / or elements), the better the performance, but the higher the accuracy drop. In addition, the larger the block size for the element (e.g., the sparse pattern), the better the performance, but the higher the accuracy drop. Therefore, Figure 1 The model training circuit module 102 selects a sparsity ratio and / or a block size for each layer of weights to balance performance and accuracy. Each layer can have a different sparsity ratio and / or block size to achieve high accuracy and low latency. The sparsity pattern can correspond to the block size and / or dimension. For example, the sparsity pattern can correspond to a block size of 8 (e.g., corresponding to a dimension of 1 times 8), a block size of 4 (e.g., corresponding to a dimension of 1 times 4), a block size of 2 (e.g., corresponding to a dimension of 1 times 2), and / or any other block size. A block can include a single weight value or a set of weight values ​​for a particular layer.

[0027] To improve accuracy, Figure 1The example model training circuit module 102 generates a mixed sparse pattern (e.g., including a sparsity ratio and a block size for a trained AI-based model) by adjusting the sparsity ratio and / or the block size for every k epochs during training. An epoch is a pass of the training data set through the AI-based model. Training the AI-based model 102 typically includes multiple (e.g., k) epochs. For a particular epoch, the model training circuit module 102 evaluates the accuracy of the AI-based model 104 based on a comparison of the output data of the AI-based model 104 with the labeled output of the training data. If the accuracy of the AI-based model 104 meets a threshold defined by a user and / or manufacturer, the model training circuit module 102 processes a different layer and / or epoch. If the accuracy of the AI-based model 104 does not meet the threshold, the model training circuit module 102 determines whether the sparsity ratio of the selected layer is higher than the threshold defined by the user and / or manufacturer, the model training circuit module 102 reduces the sparsity ratio of the layer, and the process continues to another layer and / or epoch. The model training circuit module 102 can reduce the sparsity ratio (e.g., reduce the number of zero elements of the layer) based on a user and / or manufacturer defined hyperparameter corresponding to how aggressively the sparsity ratio is reduced. The model training circuit module 102 determines that the accuracy loss ratio (e.g., also referred to as the accuracy ratio) is greater than a user-selected hyperparameter t (e.g., a threshold corresponding to a user and / or device defined loss target), the accuracy loss ratio based on a ratio of a structural loss (e.g., a loss of accuracy) to an accuracy loss of n in m. The structural loss corresponds to an accuracy loss (e.g., based on a comparison of the output of the AI-based model with the labeled output of the labeled data). The n in m sparsity pattern corresponds to n zero items in the pattern of the layer and m total items in the pattern. The example model training circuit module 102 can determine the accuracy loss of n in m (e.g., by calculating the accuracy loss and / or by identifying the accuracy loss stored in a table, register, memory, etc.). If the model training circuit module 102 determines that the accuracy loss ratio is greater than the hyperparameter selected by the user, the model training circuit module 102 changes the sparsity mode. For example, the model training circuit module 102 can reduce the block size used by the sparsity mode from an initial preset size (e.g., 1 by 8) to a smaller size (e.g., 1 by 4). A smaller block size mode will generally result in increased accuracy (e.g., impairing performance). Therefore, the model training circuit module 102 can reduce the block size until the threshold accuracy is reached. If the block size is already at the minimum size (e.g., block size or 2), the model training circuit module 102 will utilize an n out of m (e.g., 2 out of 4) sparsity mode for the layer based on user and / or manufacturer preferences. The model training circuit module 102 can utilize NVIDIA to decompose and / or adjust the weight tensor into n smaller elements in m groups.

[0028] In some examples, compared to dense AI-based models, Figure 1 The model training circuit module 102 can estimate the performance of a sparse AI-based model. For example, the model training circuit module 102 can compare a dense GEMM with a 1×4 sparse core that loads an activation matrix before a fused multiply-add (FMA) operation. In such an example, the sparse core will take 132 cycles for each FMA (e.g., 4*(3*6+3*1)+16*5, because there are four cycle operations corresponding to 3 cycles, 6 cycles, 3 cycles, and 1 cycle and 16 FMA instructions each including 5 cycles), and the dense operation reuses the activation block for FMA and the load overhead can be ignored, resulting in 80 cycles (e.g., 16*5). Therefore, the model training circuit module 102 can determine that for an example sparsity ratio of 90%, the maximum performance of the 1×4 core will be 6.06 (e.g., 100 / (100-90)*(80 / 132)). The 6.06 ratio corresponds to an upper limit of performance (eg, 6.06 times better performance than a conventional dense AI-based model). The training circuit module 102 determines the performance ratio to identify an upper limit of performance gain so that performance can be improved during training of the AI-based model 104.

[0029] In addition, the model training circuit module 102 encodes the sparse AI-based model 102 using a dynamic sparse coding scheme. Conventional encoding schemes utilize a fixed tile size for each row of the weight tensor for encoding and later performing FMA operations. However, conventional encoding schemes may result in poor computation-to-load ratios associated with FMA operations, which results in poor hardware efficiency. In order to improve hardware efficiency, the model training circuit module 102 increases the computation-to-load ratio by compressing several blocks in several columns for each row of the weight tensor to minimize load operations and maximize dot product operations. For example, the model training circuit module 102 calculates the optimal tile size for each row when encoding to improve, increase and / or maximize the computation-to-load ratio of the rows of the weight tensor. The model training circuit module 102 may try different tile sizes for the rows, calculate the computation-to-load ratios for different configurations, and select the tile size configuration based on the highest computation-to-load ratio to improve hardware efficiency. The following is combined with Figure 6 The following is an example of dynamic encoding. Figure 2 An example NN trainer is further described.

[0030] Figure 1The example AI-based model 104 includes an example system memory 106. The example system memory 106 stores encoded weights (e.g., encoded weight vectors or matrices) associated with specific neurons generated for the example model training circuit module 102. During implementation, the AI-based model 104 accesses the stored encoded weight vectors and transmits them to the corresponding neurons 110a-c to be applied to the activation data.

[0031] Figure 1 The example neurons 110a-c are structured in layers. As further described below, the neurons 110a-c are implemented by processing elements that include or communicate with MAC processing elements. The example neurons 110a-c receive input / activation data (e.g., structured as vectors / matrices), apply weights (e.g., structured as vectors / matrices) to the input / activation data to generate outputs (e.g., structured as vectors / matrices). The MAC PE can perform a multiplication and accumulation process on the activations and corresponding weights.

[0032] In training Figure 1 After the AI-based model 104 is generated, the example model training circuit module 102 deploys the encoded weight data (e.g., transmitted via one or more network communications) to the example computing device 114 via the network 112. The example network 112 is a system of interconnected systems that exchange data. The example network 112 can be implemented using any type of public or private network, such as but not limited to the Internet, a telephone network, a local area network (LAN), a cable network, and / or a wireless network. In order to enable communication via the network 112, the example AI-based model generation circuit module 100 includes a communication interface that implements a connection to an Ethernet, a digital subscriber line (DSL), a telephone line, a coaxial cable, or any wireless connection, etc. In some examples, the AI-based model generation circuit module 100 and the example computing device 114 are connected via an example network 116.

[0033] Figure 1 The example computing device 114 obtains the encoded weight data from the example AI-based model generation circuit module 110 via the network 112. The example computing device 114 can decode the encoded weight data based on dynamic sparse coding technology to locally implement the AI-based model at the computing device to perform one or more tasks.

[0034] Figure 2 yes Figure 1The example model training circuit module 102 includes an example interface circuit module 200, an example training data database 202, an example user interface 204, an example comparator 206, an example sparsity ratio converter circuit module 208, an example block mode converter circuit module 210, an example performance estimation circuit module 212, an example calculation and load calculation circuit module 214, an example encoding circuit module 216, and an example core generation circuit module 218.

[0035] Figure 2 The example interface circuit module 200 obtains (e.g., receives, accesses, etc.) encoded weights (e.g., a vector / matrix of weights) from the example system memory 106 and deploys (e.g., sends using one or more network communications) the encoded weights to the computing device 114 via the network 112. As described above, the weights are configured based on training to configure the AI-based model 104 to perform a task (e.g., classify input data). In addition, the example interface circuit module 200 can obtain (e.g., receive, access, etc.) training data for training the example AI-based model 104 via network communications, and store the obtained training data in the example training data database 202. The example interface circuit module also stores the encoded weight information in the example training data database 202 after the AI-based model 104 is trained. Figure 1 The example training data database 202 may be implemented by a storage device, a buffer, a memory, a register, etc. to store the training data.

[0036] Figure 2 The example user interface 204 obtains AI-based model characteristics and / or hyperparameters from a user. For example, a user can interact with the user interface 204 to provide characteristics and / or hyperparameters related to the number of layers of the AI-based model, the number and / or structure of neurons in the AI-based model, the number of epochs for training, a threshold and / or desired accuracy of the AI-based model, a threshold sparsity ratio for a layer, the number and / or type of block patterns to be utilized, a threshold block size, and / or any other data related to the generation of a sparse AI-based model.

[0037] Figure 2The example comparator 206 (also referred to as a comparator circuit) compares the properties of the AI-based model 104 to one or more thresholds as part of the training process. For example, the comparator 206 may compare (a) the accuracy of the AI-based model to an accuracy threshold, (b) the sparsity ratio of one or more layers to a sparsity ratio threshold, and / or (c) the block size of the block mode to a block size ratio. The comparator 206 may access the one or more thresholds from the user via the example user interface 204. Other components of the example model training circuit module 102 utilize the results / outputs of the comparator 206 to determine whether and / or how to adjust the sparsity ratio and / or block mode of the layer to generate a trained sparse AI-based model that balances accuracy and performance. In addition, the comparator 206 determines that the accuracy loss ratio based on the ratio of the structural loss (e.g., accuracy loss) to the accuracy loss of n in m is greater than a user-selected hyperparameter t (e.g., a threshold corresponding to a loss target defined by the user and / or the device). The comparator 206 may compare the accuracy loss ratio to the hyperparameter to determine whether the sparsity mode should be changed for a particular layer.

[0038] When the accuracy of the AI-based model 104 does not satisfy the threshold and the sparsity ratio of the layer is higher than the threshold, Figure 2 An example sparse ratio converter circuit module 208 adjusts the sparsity ratio (e.g., how many weights for a particular layer should correspond to zero). For example, the sparse ratio converter circuit module 208 can reduce the sparsity ratio by reducing the number of zero weights for the layer. For example, to increase sparsity, one of the hyperparameters (e.g., corresponding to pruning, learning rate, etc.) that causes the sparsity to increase can be adjusted (e.g., reducing a threshold). To reduce the sparse ratio, one or more of the hyperparameters can be adjusted to reduce the sparsity.

[0039] When the accuracy of the AI-based model 104 does not meet the threshold, the sparsity ratio of the layer is higher than the threshold, and the current block size of the layer is higher than the threshold size, Figure 2 The example block mode converter circuit module 210 adjusts the block mode ratio (e.g., 1 by 8, 1 by 4, 1 by 2, n in m, etc.). For example, if the initial block mode of a layer is set to 1 by 16, the block mode converter circuit module 210 may change the block mode to a smaller block mode (e.g., 1 by 8). If the block size of a layer is not above a threshold size (e.g., 1×2 or two blocks), the block mode converter circuit module 210 may utilize the n in m block mode for the layer.

[0040] Figure 2The example performance estimation circuit module 212 estimates the performance of the AI-based model 104. For example, the performance estimation circuit module 212 can estimate the performance of the core by performing operations such as: (1) vmovdqu8 (e.g., move unaligned packed inter-frame values), vbroadcasti32x4 (e.g., broadcast 4 packed 32-bit integers from alpha to all elements of dst), vpermt2d (e.g., full permute from two tables covering one table), vpshufb (e.g., shuffle bits from a quadword element into a mask using a byte index) (repeated 4 times), and / or (2) vpdpbusd (e.g., multiply and add unsigned and signed bytes) (repeated 16 times). In some examples, Figure 1 The performance estimation circuit module 212 can estimate the performance of a sparse AI-based model compared to a dense AI-based model. For example, the performance estimation circuit module 212 can compare a dense GEMM with a 1 by 4 sparse core that loads an activation matrix before a fused multiply-add (FMA) operation. In such an example, the sparse core will take 132 cycles (e.g., 4*(3*6+3*1)+16*5) for each FMA, and the dense operation reuses the activation block for FMA and the loading overhead can be ignored, resulting in 80 cycles (e.g., 16*5). Therefore, the model training circuit module 102 can determine that for an example sparsity ratio of 90%, the maximum performance of the 1 by 4 core will be 6.06 (e.g., 100 / (100-90)*(80 / 132)). The 6.06 ratio corresponds to an upper limit of performance (e.g., the performance is 6.06 times better than a conventional dense AI-based model).

[0041] As part of the dynamic sparse coding process, Figure 2The example computation and load calculation circuit module 214 determines the computation and load ratio for a particular row of the weight tensor. As described above, the higher the computation and load ratio, the better the hardware efficiency when applying weights to activation values. Therefore, the computation and load calculation circuit module 214 identifies different groupings / slices of weight elements in the row to identify which grouping will result in the highest computation and load calculation. For example, to determine the number of computations, the computation and load calculation circuit module 214 multiplies the number of elements in the activation block corresponding to the activation tensor / matrix (e.g., the slice size used for encoding) by the number of non-zero elements in the weight block corresponding to the weight matrix of the determined slice size (e.g., the slice size used for encoding). In addition, to determine the number of loads, the computation and load calculation circuit module 214 adds the number of elements in the activation block corresponding to the activation tensor / matrix (e.g., the slice size used for encoding) to the number of non-zero elements in the weight block corresponding to the weight matrix of the determined slice size (e.g., the slice size used for encoding). The computation and load calculation circuit module 214 determines the computation and load ratio by dividing the number of computations (e.g., computation = weight elements * activation elements) by the number of loads. The computation and load calculation circuit module 214 determines the computation and load for each of the different configurations of slices of the columns of the weight tensor, and identifies the configuration that results in the highest computation and load configuration.

[0042] Figure 2 The example encoding circuit module 216 encodes and / or compresses the weight vector / matrix based on the tile size selected by the example calculation and load calculation circuit module 214. The encoding circuit module 216 can encode the weight / vector matrix by using one or more sparse coding / compression algorithms to represent the terms by the strong weights (based on the selected tile size) of a relatively small set of neurons.

[0043] Figure 2 An example core generation circuit module 218 generates a core corresponding to a specific weight based on the dynamic sparse coding result. Because there may be different slice sizes / configurations for different rows of the weight tensor / matrix (for example, some slice sizes are smaller and other slice sizes are larger), the core generation circuit module 218 generates a core for each sparse weight to improve the efficiency of different sparse matrices. Therefore, the core is used to identify and / or promote the use of sparse weights. The core (also called the calculation core) is used during the reasoning process. For example, if the inference is c=a+b, the addition and load (a, b) / store (c) processes form the core and are called each time for a, b, and c. The core generation circuit module can use any core generation technique (for example, using xbyak, just-in-time compilation, ahead of time, etc.) to generate the core.

[0044] Although in Figure 2 The implementation is shown in Figure 1The exemplary method of model training circuit module 102 is as follows, but Figure 2 One or more of the elements, processes and / or devices shown in the example may be combined, divided, rearranged, omitted, eliminated and / or implemented in any other manner. In addition, the example interface circuit module 200, the example user interface 204, the example comparator 206, the example sparse ratio converter circuit module 208, the example block mode converter circuit module 210, the example performance estimation circuit module 212, the example calculation and load calculation circuit module 214, the example encoding circuit module 216, the example core generation circuit module 218 and / or more generally Figure 1 The example model training circuit module 102 can be implemented by hardware alone, or by hardware combined with software and / or firmware. Thus, for example, Figure 1 Any of the example interface circuit module 200, the example user interface 204, the example comparator 206, the example sparse ratio converter circuit module 208, the example block mode converter circuit module 210, the example performance estimation circuit module 212, the example calculation and load calculation circuit module 214, the example encoding circuit module 216, the example core generation circuit module 218 and / or more generally the example model training circuit module 102 can be implemented by a processor circuit module, (multiple) analog circuits, (multiple) digital circuits, (multiple) logic circuits, (multiple) programmable processors, (multiple) programmable microcontrollers, (multiple) graphics processing units (GPUs), (multiple) digital signal processors (DSPs), (multiple) application specific integrated circuits (ASICs), (multiple) programmable logic devices (PLDs) and / or (multiple) field programmable logic devices (FPLDs) (such as multiple field programmable gate arrays (FPGAs)). In addition, Figure 1 The example model training circuit module 102 may include in addition to Figure 2 In addition to or in lieu of those shown in Figure 2 One or more of the elements, processes and / or devices shown in the drawings, and / or may include more than one or all of the elements, processes and devices shown.

[0045] Figure 3-Figure 5 A flowchart representing example machine readable instructions is shown in FIG. 1 , which may be executed to configure a processor circuit module to implement Figure 2 The model training circuit module 102. The machine-readable instructions may be one or more executable programs or (one or more) parts of executable programs for execution by a processor circuit module, such as the following combination Figure 7 The processor circuit module 712 shown in the example processor platform 700 discussed below and / or in conjunction with Figure 8 and / or Fig. 9Example processor circuit modules discussed. The program may be embodied in software stored on one or more non-transitory computer-readable storage media, such as a compact disk (CD), a floppy disk, a hard disk drive (HDD), a solid-state drive (SSD), a digital versatile disk (DVD), a Blu-ray disk, a volatile memory (e.g., any type of random access memory (RAM), etc.) or a non-volatile memory associated with a processor circuit module located in one or more hardware devices (e.g., electrically erasable programmable read-only memory (EEPROM), FLASH memory, HDD, SSD, etc.), but the entire program and / or portions thereof may alternatively be executed by one or more hardware devices other than the processor circuit module and / or embodied in firmware or dedicated hardware. The machine-readable instructions may be distributed across multiple hardware devices and / or executed by two or more hardware devices (e.g., a server and a client hardware device). For example, a client hardware device may be implemented by an endpoint client hardware device (e.g., a hardware device associated with a user) or an intermediate client hardware device (e.g., a radio access network (RAN) gateway that may facilitate communication between a server and an endpoint client hardware device). Similarly, non-transitory computer-readable storage media may include one or more media located in one or more hardware devices. Figure 3-Figure 5 The flowchart shown in describes an example program, but many other methods of implementing the example model training circuit module 102 may be used instead. For example, the order of execution of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the boxes may be implemented by one or more hardware circuits (e.g., processor circuit modules, discrete and / or integrated analog and / or digital circuit modules, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are constructed to perform corresponding operations without executing software or firmware. The processor circuit modules may be distributed in different network locations and / or located locally in one or more hardware devices (e.g., single-core processors (e.g., single-core central processing units (CPUs)), multi-core processors in a single machine (e.g., multi-core CPUs, XPUs, etc.), multiple processors distributed on multiple servers in a server rack, multiple processors distributed on one or more server racks, CPUs and / or FPGAs in the same package (e.g., the same integrated circuit (IC) package or two or more separate housings, etc.).

[0046] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, a packaged format, etc. The machine-readable instructions as described herein may be stored as data or data structures (e.g., as part of instructions, codes, code representations, etc.) that can be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers) located at the same or different locations of a network or a collection of networks (e.g., in the cloud, in edge devices, etc.). The machine-readable instructions may need to be installed, modified, adapted, updated, combined, supplemented, configured, decrypted, decompressed, decapsulated, distributed, redistributed, compiled, etc., so that they can be directly read, interpreted, and / or executed by a computing device and / or other machine. For example, the machine-readable instructions may be stored in multiple parts that are individually compressed, encrypted, and / or stored on separate computing devices, where the parts, when decrypted, decompressed, and / or combined, form a set of machine-executable instructions that implement one or more operations that can together form a program such as described herein.

[0047] In another example, the machine-readable instructions may be stored in a state where they can be read by a processor circuit module, but a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., may need to be added in order to execute the machine-readable instructions on a particular computing device or other device. In another example, the machine-readable instructions may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before the machine-readable instructions and / or corresponding programs can be executed in whole or in part. Therefore, as used herein, a machine-readable medium may include machine-readable instructions and / or programs regardless of the specific format or state of the machine-readable instructions and / or programs when they are stored or otherwise at rest or in transit.

[0048] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, the machine-readable instructions may be represented using any of the following languages: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0049] As mentioned above, Figure 3-Figure 5The example operations of may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on one or more non-transitory computer and / or machine readable media, such as optical storage devices, magnetic storage devices, HDDs, flash memory, read-only memory (ROM), CDs, DVDs, caches, any type of RAM, registers, and / or any other storage device or storage disk in which information is stored for any duration (e.g., for an extended period of time, permanently, temporarily, for temporary buffering, and / or for caching of information). As used herein, the terms non-transitory computer readable medium, non-transitory computer readable storage medium, non-transitory machine readable medium, and non-transitory machine readable storage medium are expressly defined to include any type of computer readable storage device and / or storage disk, and to exclude propagating signals and to exclude transmission media. As used herein, the terms "computer readable storage device" and "machine readable storage device" are defined to include any physical (mechanical and / or electrical) structure to store information, but to exclude propagating signals and to exclude transmission media. Examples of computer-readable storage devices and machine-readable storage devices include any type of random access memory, any type of read-only memory, solid-state memory, flash memory, optical disks, magnetic disks, disk drives, and / or redundant arrays of independent disks (RAID) systems. As used herein, the term "device" refers to a physical structure, such as a mechanical and / or electrical device, hardware, and / or circuit module, that may or may not be configured by computer-readable instructions, machine-readable instructions, etc. and / or manufactured to execute computer-readable instructions, machine-readable instructions, etc.

[0050] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim employs any form of "include" or "comprise" (e.g., comprises, includes, comprising, including, having, etc.) as a preamble or within any type of claim recitation, it should be understood that additional elements, terms, etc. may be present without exceeding the scope of the corresponding claim or recitation. As used herein, when the phrase "at least" is used as a transitional term in the preamble of a claim, it is open-ended in the same manner as the terms "include" and "comprising" are open-ended. When used, for example, in a form such as A, B, and / or C, the term "and / or" refers to any combination or subset of A, B, and C, such as (1) A alone, (2) B alone, (3) C alone, (4) A and B, (5) A and C, (6) B and C, or (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A and B" is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A or B" is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. As used herein in the context of describing the implementation or execution of processes, instructions, actions, activities, and / or steps, the phrase "at least one of A and B" is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B. Similarly, as used herein in the context of describing the implementation or execution of processes, instructions, actions, activities, and / or steps, the phrase "at least one of A or B" is intended to refer to implementations that include any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0051] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude a plurality. As used herein, the term "a" or "an" object refers to one or more of the object. The terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. Furthermore, although listed separately, multiple devices, elements, or method actions may be implemented by, for example, the same entity or object. In addition, although individual features may be included in different examples or claims, these features may be combined, and inclusion in different examples or claims does not mean that a combination of features is not feasible and / or advantageous.

[0052] Figure 3 is a flow diagram representative of example machine readable instructions and / or example operations 300 that may be executed and / or instantiated by a processor circuit module to generate and encode a sparse AI-based model. Figure 3 The machine-readable instructions and / or operations 300 of the embodiment begin at block 302, where the user interface 204 and / or the interface circuit module 200 obtains a method for generating (e.g., training) an AI-based model (e.g., Figure 1 For example, if the instruction comes from a user, the user interface 204 will obtain the instruction, and if the instruction comes from another component or device, the interface circuit module 200 will obtain the instruction.

[0053] At block 304, the example user interface 204 determines parameters of the AI-based model. The parameters may include configurations, hyperparameters, thresholds, target accuracy, etc. that a user may select to configure and / or train the AI-based model. At block 306, the example interface circuit module 200 obtains training data from the example training data database 202 to apply to the AI-based model 104 to train the AI-based model 104. At block 308, the example model training circuit module 102 trains a sparsity model based on the training data, as described below in conjunction with Figure 4 Further described.

[0054] At block 310, the example performance estimation circuit module 212 estimates the performance of the trained sparsity model, as described above in conjunction with Figure 2 The training circuit module 102 determines the performance ratio to identify an upper limit of the performance gain so that the performance can be improved during the training of the AI-based model 104. At block 312, the example model training circuit module 102 performs dynamic sparse encoding of the sparse model, as described below in conjunction with Figure 5At block 313, the example core generation circuit module 218 generates a core for each sparse weight of the trained AI-based model. At block 314, the example interface circuit module 200 stores the encoded weight information in Figure 1 In the example system memory 106 of FIG. 316 , the example interface circuit module 200 accesses the stored encoded weight information and transmits the weight information to the example system memory 106 of FIG. 316 . Figure 1 The network 112 deploys (eg, transmits) the encoded sparsity model (eg, encoded weight information) to the computing device 114 .

[0055] Figure 4 is a flow chart representing example machine readable instructions and / or example operations 308 that may be executed and / or instantiated by a processor circuit module to train a sparsity model based on training data. During training (blocks 402-424) for each epoch, execute and / or run Figure 4 The machine readable instructions and / or operations 308 of the embodiment of the present invention are to determine a hybrid parsing mode for a layer of the AI-based model 104. At block 404, the comparator 206 selects a layer of the AI-based model to apply the block sparsity.

[0056] At box 406, the example comparator 206 determines the accuracy of the AI-based model 104 based on the output of the model and / or the output of the layer and the output included in the labeled data. For example, the comparator 206 determines the accuracy of the AI-based model 104 by comparing the output of the AI-based model 104 with the expected output of the labeled data (e.g., determining the difference) to see how close the output is to determine the accuracy. At box 408, the example comparator 206 compares the accuracy (or the drop in accuracy) with a threshold. If the comparator 206 determines that the accuracy of the AI-based model 104 meets (e.g., is higher than) the threshold (box 408: yes), control continues to box 420. If the comparator 206 determines that the accuracy of the AI-based model 104 does not meet (e.g., is lower than) the threshold (box 408: no), the example comparator 206 determines the sparse ratio of the selected layer by determining the number of non-zero elements in the layer and the total number of elements of the layer (box 409). At block 410 , the example comparator 206 determines whether the sparsity ratio of the selected layer is above a threshold (eg, a sparsity ratio threshold).

[0057] If the example comparator 206 determines that the sparse ratio is above the threshold (block 410: yes), the example sparse ratio converter circuit module 208 changes the sparse ratio (block 412), and control continues to block 420. For example, the sparse ratio converter circuit module may reduce the sparse ratio to improve the accuracy of the AI-based model 104, as described above in conjunction with Figure 2Further described. If the example comparator 206 determines that the sparse ratio is not higher than the threshold (box 410: No), the example comparator 206 determines the accuracy loss ratio (box 413). The model training circuit module 102 determines the accuracy loss ratio based on the ratio of the structural loss (e.g., accuracy loss) and the accuracy loss of the n-mode in m. At box 414, the example comparator 206 determines whether the accuracy loss ratio is greater than a hyperparameter t (e.g., a threshold corresponding to a loss target defined by a user and / or device). If the example comparator 206 determines that the accuracy loss ratio is not greater than the hyperparameter (box 414: No), control continues to box 420.

[0058] If the example comparator 206 determines that the accuracy loss ratio is greater than the hyperparameter (box 414: yes), the example comparator 206 determines whether the block size for the selected layer is less than a threshold (e.g., a block size threshold) (box 415). The block size can be initialized to a first size (e.g., 1 by 16), and the block size is reduced and / or changed to a different size to improve accuracy. Therefore, if the example comparator 206 determines that the block size is not less than the threshold (box 415: no), the example block mode converter circuit module 210 changes the block mode (e.g., size) for the layer (e.g., to a smaller size) (box 416), and control continues to box 420. If the example comparator 206 determines that the block size is less than the threshold (box 415: yes), the example block mode converter circuit module 210 changes the block mode to utilize the n-in-m mode (box 418). As described above, the n-in-m mode has higher accuracy at the cost of performance.

[0059] At block 420, the example comparator 206 determines whether there are additional layers to be processed. If the example comparator 206 determines that there are additional layers to be processed (block 422: yes), the example comparator 206 selects a subsequent layer of the AI-based model 102 (block 422), and control returns to block 404. If the example comparator 206 determines that there are no additional layers to be processed (block 422: no), additional layers are processed and / or control returns to block 404. Figure 3 Frame 310.

[0060] Figure 5 is a flow diagram representative of example machine readable instructions and / or example operations 312 that may be executed and / or instantiated by a processor circuit module to perform dynamic sparse encoding of a sparsity model. Figure 4 The machine readable instructions and / or operations 312 begin at block 502 where the example calculation and load calculation circuit module 214 selects a row of a weight tensor.

[0061] At block 504, the example compute and load calculation circuit module 214 selects a compute and load ratio for the selected row. For example, the compute and load calculation circuit module 214 determines the compute and load ratio by dividing the number of computations required to multiply the activation slice by the weight row by the number of loads required to multiply the activation slice by the weight row. To determine the number of computations required, the compute and load calculation circuit module 214 multiplies the number of elements in the activation slice corresponding to the activation tensor / matrix by the number of non-zero elements in the row of the weight matrix. To determine the number of loads required, the compute and load calculation circuit module 214 adds the number of elements in the activation slice corresponding to the activation tensor / matrix to the number of non-zero elements in the row of the weight matrix.

[0062] At box 506, the example compute and load calculation circuit module 214 determines different groupings and / or slicings of rows. For example, the compute and load calculation circuit module 214 may slice / group rows into different configurations of slices / groups (e.g., converting a 1×16 row into four 1×4 slices, one 1×8 slice and two 1×4 slices, eight 1×2 slices, and / or any other configuration of slices). At box 508, the example compute and load calculation circuit module 214 determines the compute and load ratios for the different groupings and / or slicings. At box 510, the example compute and load calculation circuit module 214 selects the row or grouping / slicing that results in the highest compute and load ratio, thereby resulting in the highest hardware efficiency when encoding. At box 512, the example encoding circuit module 216 encodes the rows of the weight tensor based on the selected row or grouping / slicing configuration.

[0063] At block 514, the example calculation and load calculation circuit module 214 determines whether there is another row of the weight tensor / matrix to be processed / encoded. If the example calculation and load calculation circuit module 214 determines that there is another row to be processed (block 514: yes), the calculation and load calculation circuit module 214 selects a subsequent row (block 516) and control returns to block 504. If the example calculation and load calculation circuit module 214 determines that there is no another row to be processed (block 514: no), control returns to Figure 3 Frame 314.

[0064] Figure 6 An example of dynamic sparse coding disclosed herein is shown. Figure 6 The example includes an example activation tensor 600 (matrix A) and an example weight tensor 602 (matrix B). The AI-based model training circuit module 102 changes the size of the slices for each row based on density (e.g., how many non-zero blocks are included in a row), with denser rows having smaller slices. Figure 6In the example, the activation tensor 600 is tiled into 4 by 1 slices along M columns. For the first row of the weight tensor 602, the example AI-based model training circuit module 102 determines the computation to load ratio as 12 / 7 and selects the entire row for encoding. For the fifth row of the weight tensor 602, the AI-based model training circuit module 102 determines the computation to load of the first slice as 16 / 8 (e.g., (4 of the M*4 non-zero elements in the first slice from the fifth row) / (4 of the M+4 non-zero elements in the first slice from the fifth row)), and determines the computation to load of the second slice as 12 / 7 (e.g., (4 of the M*3 non-zero elements in the first slice from the fifth row) / (4 of the M+3 non-zero elements in the first slice from the fifth row)). In some examples, the AI-based model training circuit module 102 may further change the size of the tiles based on the availability of registers for generating outputs.

[0065] Figure 7 is constructed to execute Figure 3-Figure 5 Instructions to achieve Figure 1 and / or Figure 2 The processor platform 700 is a block diagram of an example processor platform 700 of an example model training circuit module 102. The processor platform 700 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet computer (such as an iPad) TM )), a personal digital assistant (PDA), an Internet appliance, or any other type of computing device.

[0066] The processor platform 700 of the illustrated example includes a processor 712. The processor 712 of the illustrated example is hardware. For example, the processor 712 may be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers from any desired family or manufacturer. The hardware processor may be a semiconductor-based (e.g., silicon-based) device. In this example, the processor 712 implements Figure 2 At least one of the example user interface 204, the example comparator 206, the example sparse ratio converter circuit module 208, the example block mode converter circuit module 210, the example performance estimation circuit module 212, the example calculation and load calculation circuit module 214, the example encoding circuit module 216 and the example core generation circuit module 218.

[0067] The processor 712 of the illustrated example includes a local memory 713 (e.g., a cache). Figure 7 In the example, local storage 713 implements Figure 2The example training data database 202 of the illustrated example is provided. The processor 712 of the illustrated example communicates with a main memory including a volatile memory 714 and a non-volatile memory 716 via a bus 718. The volatile memory 714 may be comprised of a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory The example local memory 713, the example volatile memory 714, and / or the example non-volatile memory 716 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 714, 716 is controlled by a memory controller. The example local memory 713, the example volatile memory 714, and / or the example non-volatile memory 716 may be implemented by Figure 1 Any of the example volatile memory 714, the example non-volatile memory 716, and / or the example mass storage device 728 may implement Figure 1 An example system memory 106 and / or Figure 2 The example training data database 202 of FIG.

[0068] The processor platform 700 of the illustrated example also includes an interface circuit 720. The interface circuit 720 may be implemented by any type of interface standard, such as an Ethernet interface, a universal serial bus (USB), and interface, Near Field Communication (NFC) interface, and / or PCI Express interface. Figure 7 In the example, interface 720 implements Figure 2 An example interface circuit module 200 is provided.

[0069] In the example shown, one or more input devices 722 are connected to the interface circuit 720. The input devices 722 allow a user to enter data and / or commands into the processor 712. The input devices may be implemented by, for example, an audio sensor, a microphone, a camera (still or video), a keyboard, buttons, a mouse, a touch screen, a trackpad, a trackball, and / or a voice recognition system.

[0070] One or more output devices 724 are also connected to the interface circuit 720 of the illustrated example. The output device 724 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switch (IPS) display, a touch screen, etc.), a tactile output device, and / or a speaker. Therefore, the interface circuit 720 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0071] The interface circuitry 720 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any kind of computing device) via a network 726. Communications may be via, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a line-of-site wireless system, a cellular system, etc.

[0072] The processor platform 700 of the illustrated example also includes one or more mass storage devices 728 for storing software and / or data. Examples of such mass storage devices 728 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0073] Figure 3-Figure 5 The machine-executable instructions 732 may be stored in the mass storage device 728, in the volatile memory 714, in the non-volatile memory 716, and / or on a removable, non-transitory computer-readable storage medium such as a CD or DVD.

[0074] Figure 8 yes Figure 7 712. In this example, Figure 7 The processor circuit module 712 is implemented by the microprocessor 800. For example, the microprocessor 900 can implement a multi-core hardware circuit module, such as a CPU, a DSP, a GPU, an XPU, etc. Although it can include any number of example cores 802 (for example, 1 core), the microprocessor 800 of this example is a multi-core semiconductor device including N cores. The cores 802 of the microprocessor 800 can operate independently or can cooperate to execute machine-readable instructions. For example, the machine code corresponding to a firmware program, an embedded software program, or a software program can be executed by one of the cores 802, or can be executed by multiple of the cores 802 at the same or different times. In some examples, the machine code corresponding to the firmware program, the embedded software program, or the software program is split into threads and executed in parallel by two or more of the cores 802. The software program can be executed in parallel by Figure 3-Figure 5 The flowcharts correspond to a portion or all of machine-readable instructions and / or operations.

[0075] The core 802 can communicate via an example bus 804. In some examples, the bus 804 can implement a communication bus to implement communications associated with one (or more) of the cores 802. For example, the bus 804 can implement at least one of an inter-integrated circuit (I2C) bus, a serial peripheral interface (SPI) bus, a PCI bus, or a PCIe bus. Additionally or alternatively, the bus 804 can implement any other type of computing or electrical bus. The core 802 can obtain data, instructions, and / or signals from one or more external devices via an example interface circuit module 806. The core 802 can output data, instructions, and / or signals to one or more external devices via the interface circuit module 806. Although the core 802 of this example includes an example local memory 820 (e.g., a level 1 (L1) cache that can be split into an L1 data cache and an L1 instruction cache), the microprocessor 800 also includes an example shared memory 810 (e.g., a level 2 (L2 cache)) that can be shared by the cores for high-speed access to data and / or instructions. Data and / or instructions may be transferred (e.g., shared) by writing data and / or instructions to shared memory 810 and / or reading data and / or instructions from shared memory 810. Local memory 820 of each of cores 802 and shared memory 810 may be a plurality of levels of cache memory and main memory (e.g., Figure 7 Cache memory is part of a hierarchy of storage devices (main memory 714, 716). Generally, higher level memory in the hierarchy exhibits lower access times and has smaller storage capacity than lower level memory. Changes to various levels of the cache hierarchy are managed (e.g., coordinated) by a cache coherence policy.

[0076] Each core 802 may be referred to as a CPU, DSP, GPU, etc., or any other type of hardware circuit module. Each core 802 includes a control unit circuit module 814, an arithmetic and logic (AL) circuit module (sometimes referred to as an ALU) 816, a plurality of registers 818, an L1 cache 820, and an example bus 822. Other structures may exist. For example, each core 802 may include a vector unit circuit module, a single instruction multiple data (SIMD) unit circuit module, a load / store unit (LSU) circuit module, a branch / jump unit circuit module, a floating point unit (FPU) circuit module, etc. The control unit circuit module 814 includes a semiconductor-based circuit configured to control (e.g., coordinate) data movement within the corresponding core 802. The AL circuit module 816 includes a semiconductor-based circuit configured to perform one or more mathematical and / or logical operations on data within the corresponding core 802. The AL circuit module 816 of some examples performs integer-based operations. In other examples, the AL circuit module 816 also performs floating-point operations. In other examples, the AL circuit module 816 may include a first AL circuit module that performs integer-based operations and a second AL circuit module that performs floating-point operations. In some examples, the AL circuit module 816 may be referred to as an arithmetic logic unit (ALU). The register 818 is a semiconductor-based structure for storing data and / or instructions, such as the results of one or more of the operations performed by the AL circuit module 816 of the corresponding core 802. For example, the register 818 may include a vector register, a SIMD register, a general register, a flag register, a segment register, a machine-specific register, an instruction pointer register, a control register, a debug register, a memory management register, a machine check register, etc. The register 818 may be arranged in a manner such as Figure 8 Alternatively, registers 818 may be organized in any other arrangement, format, or structure, including an arrangement, format, or structure distributed throughout core 802 to reduce access time. Bus 820 may implement at least one of an I2C bus, an SPI bus, a PCI bus, or a PCIe bus.

[0077] Each core 802 and / or more generally, the microprocessor 800 may include additional and / or alternative structures to those shown and described above. For example, there may be one or more clock circuits, one or more power supplies, one or more power gates, one or more cache home agents (CHA), one or more aggregation / common grid stops (CMS), one or more shifters (e.g., barrel shifters) and / or other circuit modules. The microprocessor 800 is a semiconductor device that is manufactured to include many transistors that are interconnected to implement the above structure in one or more integrated circuits (ICs) contained in one or more packages. The processor circuit module may include one or more accelerators and / or collaborate with one or more accelerators. In some examples, the accelerator is implemented by a logic circuit to perform certain tasks faster and / or more efficiently than a general-purpose processor can accomplish. Examples of accelerators include ASICs and FPGAs, such as those discussed herein. A GPU or other programmable device may also be an accelerator. The accelerator may be on a processor circuit module, in the same chip package as the processor circuit module, and / or in one or more independent packages separated from the processor circuit module.

[0078] Fig. 9 yes Figure 7 900. The processor circuit module 712 is implemented by an FPGA circuit module 900. The FPGA circuit module 900 can be used, for example, to perform operations that can be performed by executing corresponding machine-readable instructions. Figure 8 However, once configured, the FPGA circuit module 900 instantiates machine-readable instructions in hardware and can therefore typically perform operations faster than a general-purpose microprocessor executing corresponding software.

[0079] More specifically, Figure 8 The microprocessor 800 (which is a general-purpose device that can be programmed to perform the Figure 3-Figure 5 Compared to a flowchart that represents some or all of the machine-readable instructions, but whose interconnections and logic circuit modules are fixed once they are manufactured, Fig. 9 The example FPGA circuit module 900 includes interconnects and logic circuit modules, which can be configured and / or interconnected in different ways after manufacturing to instantiate, for example, Figure 3-Some or all of the machine-readable instructions represented by the flowchart of Figure 900. In particular, FPGA 900 can be considered as an array of logic gates, interconnects, and switches. The switches can be programmed to change how the logic gates are interconnected by the interconnects, effectively forming one or more dedicated logic circuits (unless and until FPGA circuit module 900 is reprogrammed). The configured logic circuits enable the logic gates to cooperate in different ways to perform different operations on the data received by the input circuit module. These operations can correspond to Figure 3-Figure 5 Thus, the FPGA circuit module 900 can be constructed to effectively Figure 3-Figure 5 Some or all of the machine-readable instructions of the flowchart of the embodiment of the present invention are instantiated as dedicated logic circuits to perform operations corresponding to those software instructions in a dedicated manner similar to an ASIC. Therefore, the FPGA circuit module 900 can perform operations faster than a general-purpose microprocessor. Figure 3-Figure 5 Operations corresponding to some or all of the machine-readable instructions.

[0080] exist Fig. 9 In the example of FPGA circuit module 900, FPGA circuit module 900 is configured to be programmed (and / or reprogrammed one or more times) by an end user via a hardware description language (HDL) such as Verilog. Fig. 9 The FPGA circuit module 900 includes an example input / output (I / O) circuit module 902 to obtain and / or output data to / from an example configuration circuit module 904 and / or external hardware (e.g., external hardware circuit) 906. For example, the configuration circuit module 904 can implement an interface circuit that can obtain machine-readable instructions to configure the FPGA circuit module 900 or (multiple) portions thereof. In some such examples, the configuration circuit module 904 can obtain machine-readable instructions from a user, a machine (e.g., a hardware circuit (e.g., a programmed or dedicated circuit) that can implement an artificial intelligence / machine learning (AI / ML) model to generate instructions), etc. In some examples, the external hardware 906 can implement Figure 8 The FPGA circuit module 900 also includes an array of example logic gate circuit modules 908, a plurality of example configurable interconnects 910, and an example storage circuit module 912. The logic gate circuit modules 908 and the interconnects 910 can be configured to communicate with Figure 3-Figure 5 At least some of the machine-readable instructions correspond to one or more operations and / or other desired operations are instantiated. Fig. 9The logic gate circuit modules 908 shown in are manufactured into groups or blocks. Each block includes a semiconductor-based electrical structure that can be configured into a logic circuit. In some examples, the electrical structure includes a logic gate (e.g., an AND gate, an OR gate, a NOR gate, etc.) that provides a basic building block for the logic circuit. An electrically controllable switch (e.g., a transistor) is present in each of the logic gate circuit modules 908 so that the electrical structure and / or the logic gate can be configured to form a circuit for performing the desired operation. The logic gate circuit module 908 may include other electrical structures, such as a lookup table (LUT), a register (e.g., a flip-flop or a latch), a multiplexer, etc.

[0081] The interconnect 910 of the illustrated example is a conductive path, trace, via, etc., which may include an electrically controllable switch (e.g., a transistor) whose state can be changed by programming (e.g., using an HDL instruction language) to activate or deactivate one or more connections between one or more logic gate circuit modules 908 to program the desired logic circuit.

[0082] The storage circuit module 912 of the illustrated example is configured to store the result of one or more operations performed by the corresponding logic gates. The storage circuit module 912 may be implemented by a register, etc. In the illustrated example, the storage circuit module 912 is distributed between the logic gate circuit modules 908 to facilitate access and improve execution speed.

[0083] Fig. 9 The example FPGA circuit module 900 also includes an example dedicated operation circuit module 914. In this example, the dedicated operation circuit module 914 includes a special-purpose circuit module 916, which can be called to implement common functions to avoid the need to program these functions on site. Examples of such special-purpose modules 916 include memory (e.g., DRAM) controller circuit modules, PCIe controller circuit modules, clock circuit modules, transceiver circuit modules, memory, and multiplier-accumulator circuit modules. There may be other types of dedicated circuit modules. In some examples, the FPGA circuit module 900 may also include an example general-purpose programmable circuit module 918, such as an example CPU 920 and / or an example DSP 922. There may be additionally or alternatively other general-purpose programmable circuit modules 918 that can be programmed to perform other operations, such as GPUs, XPUs, etc.

[0084] although Figure 8 and Fig. 9 Shows Figure 7 These are two example implementations of the processor circuit module 712, but many other approaches are contemplated. For example, as described above, a modern FPGA circuit module may include an onboard CPU, such as Fig. 9 One or more of the example CPUs 920. Thus, Figure 7 The processor circuit module 712 can also be combined Figure 8 An example microprocessor 800 and Fig. 9 In some such hybrid examples, the Figure 3-Figure 5 The first portion of the machine-readable instructions represented by the flowchart may be represented by Figure 8 One or more cores 802 are executed and are executed by Figure 3-Figure 5 The second portion of the machine-readable instructions represented by the flowchart may be represented by Fig. 9 FPGA circuit module 900 executes.

[0085] In some examples, Figure 7 The processor circuit module 712 may be in one or more packages. For example, Figure 8 The processor circuit module 800 and / or Fig. 9 The FPGA circuit module 900 can be in one or more packages. In some examples, the XPU can be composed of Figure 7 The XPU may be implemented in a processor circuit module 712, which may be in one or more packages. For example, the XPU may include a CPU in one package, a DSP in another package, a GPU in yet another package, and an FPGA in yet another package.

[0086] Fig.10 The diagram is used to illustrate the Figure 7 1005 is a block diagram of an example software distribution platform 1005 for distributing software such as example computer readable instructions 732 to third parties. The example software distribution platform 1005 can be implemented by any computer server, data facility, cloud service, etc. that can store software and transfer software to other computing devices. The third party can be a customer of an entity that owns and / or operates the software distribution platform. For example, the entity that owns and / or operates the software distribution platform can be a software (such as Figure 7 The third party may be a consumer, user, retailer, OEM, etc. who purchases and / or licenses the software for use and / or resale and / or sub-license. In the example shown, the software distribution platform 1005 includes one or more servers and one or more storage devices. The storage device stores computer readable instructions 732, which may be used with the computer readable instructions 732 as described above. Figure 3-Figure 5 and Figure 7The example computer readable instructions 300, 308, 312 of the example software distribution platform 1005 correspond to one or more servers of the example software distribution platform 1005. The one or more servers of the example software distribution platform 1005 communicate with a network 1010, which may correspond to any one or more of the Internet and / or any of the example networks 726 described above. In some examples, one or more servers respond to a request to send software to a requesting party as part of a business transaction. The delivery, sale, and / or payment of a license for the software may be processed by one or more servers of the software distribution platform and / or via a third-party payment entity. The server enables a purchaser and / or a licensee to download computer readable instructions 732 from the software distribution platform 1005. For example, the computer readable instructions 732 may be downloaded from the software distribution platform 1005. Figure 7 The software corresponding to the example computer readable instructions 732 can be downloaded to the example processor platform 1000, which is used to execute the computer readable instructions 732 to implement the model training circuit module 102. In some examples, one or more servers of the software distribution platform 1005 periodically provide, send and / or force software updates (e.g., Figure 7 Example computer readable instructions 732) to ensure that improvements, patches, updates, etc. are distributed and applied to the software at the end-user device.

[0087] Example methods, apparatus, systems, and articles for performing artificial intelligence-based sparse computations based on hybrid patterns and dynamic encoding. Further examples and combinations thereof include the following: Example 1 includes an apparatus comprising a memory, computer-readable instructions, and a processor circuit module, the processor circuit module being configured to execute computer-readable instructions to: determine a hybrid sparsity pattern for a selected layer of an artificial intelligence (AI)-based model, the hybrid sparsity pattern having a sparsity ratio and a block pattern for the selected layer; in response to the sparsity ratio being above a threshold, reduce the sparsity ratio of the selected layer; and in response to the sparsity ratio being below a threshold, adjust the block pattern of the selected layer, the block pattern of the selected layer corresponding to the accuracy ratio.

[0088] Example 2 includes the apparatus of Example 1, wherein the processor circuit module is to reduce the sparsity ratio by reducing the number of zero weights for the selected layer.

[0089] Example 3 includes an apparatus as described in Example 1, wherein the processor circuit module is used to adjust the block pattern by reducing the number of elements per block in the selected layer in response to n patterns in m of the block pattern of the selected layer being greater than a threshold corresponding to a hyperparameter.

[0090] Example 4 includes the apparatus of Example 3, wherein the processor circuit module is to: in response to a block size corresponding to the block mode being less than a threshold size, adjust the block mode to an n-of-m mode.

[0091] Example 5 includes an apparatus as described in Example 1, wherein the processor circuit module is used to determine the accuracy ratio based on a comparison of the output of the AI-based model with the labeled output of the training data, wherein the output of the AI-based model is based on the training data applied to the AI-based model.

[0092] Example 6 includes an apparatus as described in Example 1, wherein the processor circuit module is used to estimate the performance of the AI-based model based on a comparison of sparse operations and dense operations corresponding to the AI-based model.

[0093] Example 7 includes an apparatus as described in Example 1, wherein the processor circuit module is used to generate a kernel for each sparse weight of the selected layer of the AI-based model.

[0094] Example 8 includes an apparatus as described in Example 7, wherein the processor circuit module is used to: dynamically sparse encode the AI-based model by selecting grouping of data in the AI-based model based on a computation to load ratio; and generate a kernel for each sparse weight based on the dynamically encoded AI-based model.

[0095] Example 9 includes a non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to at least: determine a hybrid sparsity pattern for a selected layer of an artificial intelligence (AI)-based model, the hybrid sparsity pattern having a sparsity ratio and a block pattern for the selected layer; in response to the sparsity ratio being higher than a threshold, reduce the sparsity ratio of the selected layer; and in response to the sparsity ratio being lower than a threshold, adjust the block pattern of the selected layer, the block pattern of the selected layer corresponding to the accuracy ratio.

[0096] Example 10 includes the computer-readable medium of Example 9, wherein the instructions cause one or more processors to reduce the sparsity ratio by reducing a number of zero weights for the selected layer.

[0097] Example 11 includes the computer-readable medium of Example 9, wherein the instructions cause one or more processors to adjust the block pattern by reducing the number of elements per block in the selected layer in response to n patterns of m of the block patterns of the selected layer being greater than a threshold corresponding to a hyperparameter.

[0098] Example 12 includes the computer readable medium of Example 11, wherein the instructions cause the one or more processors to adjust the block mode to an n-in-m mode in response to a block size corresponding to the block mode being less than a threshold size.

[0099] Example 13 includes the computer-readable medium of Example 9, wherein the instructions cause one or more processors to determine an accuracy ratio based on a comparison of an output of a selected layer with labeled outputs of the training data, the output of the selected layer being based on the training data applied to the AI-based model.

[0100] Example 14 includes the computer-readable medium of Example 9, wherein the instructions cause one or more processors to estimate performance of the AI-based model based on a comparison of sparse operations and dense operations corresponding to the AI-based model.

[0101] Example 15 includes the computer-readable medium of Example 9, wherein the instructions cause one or more processors to generate a kernel for each sparse weight of the selected layer of the AI-based model.

[0102] Example 16 includes a computer-readable medium as described in Example 15, wherein the instructions cause the one or more processors to: dynamically sparse encode the AI-based model by selecting a grouping of data in the AI-based model based on a computation to load ratio; and generate a kernel for each sparse weight based on the dynamically encoded AI-based model.

[0103] Example 17 includes a method comprising: determining a mixed sparsity pattern of a layer of an artificial intelligence (AI) based model by executing instructions using one or more processors, the mixed sparsity pattern having a sparsity ratio and a block pattern for the layer; if a first sparsity ratio of the first layer is higher than a threshold, reducing the first sparsity ratio of the first layer by executing instructions using one or more processors; and if a second sparsity ratio of the second layer is lower than a threshold, adjusting the block pattern of the second layer by executing instructions using one or more processors, the block pattern of the selected layer corresponding to the accuracy ratio.

[0104] Example 18 includes the method of Example 17, wherein reducing the first sparsity ratio comprises reducing a number of zero weights for the first layer.

[0105] Example 19 includes the method of Example 17, wherein the adjusting of the block pattern comprises reducing the number of elements per block in the first layer in response to n patterns of m in the block pattern of the second layer being greater than a threshold corresponding to a hyperparameter.

[0106] Example 20 includes the method of Example 19, wherein the adjusting of the block mode comprises: in response to a block size corresponding to the block mode being less than a threshold size, adjusting the block mode to an n-of-m mode.

[0107] Example 21 includes the method of Example 17, further comprising determining an accuracy ratio based on a comparison of an output of the AI-based model with a labeled output of the training data, wherein the output of the AI-based model is based on the training data applied to the AI-based model.

[0108] Example 22 includes the method of Example 17, further comprising determining an accuracy ratio based on a comparison of an output of the AI-based model with a labeled output of the training data, wherein the output of the AI-based model is based on the training data applied to the AI-based model.

[0109] Example 23 includes the method of Example 17, further comprising estimating performance of the AI-based model based on a comparison of sparse operations and dense operations corresponding to the AI-based model.

[0110] Example 24 includes the method of Example 17, further comprising generating a kernel for each sparse weight of at least one of the first layer or the second layer of the AI-based model.

[0111] Example 25 includes the method as described in Example 24, further comprising: dynamically sparse encoding the AI-based model by selecting a grouping of data in the AI-based model based on a computation to load ratio; and generating a kernel for each sparse weight based on the dynamically encoded AI-based model.

[0112] The examples disclosed herein perform artificial intelligence-based sparse computations based on hybrid modes and dynamic encoding. The examples disclosed herein result in a balance between accuracy and performance based on the configuration and / or characteristics of the AI-based model itself. In addition, dynamic encoding by slicing based on the density of the rows of the weight vector improves hardware efficiency. Therefore, the disclosed methods, devices, and articles therefore relate to one or more improvements in the functionality of the AI-based model.

[0113] Although certain exemplary methods, apparatus, and articles have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles that fully fall within the scope of the claims of this patent.

[0114] The following claims are hereby incorporated into this Detailed Description, with each claim standing on its own as a separate embodiment of the disclosure.

Claims

1. A device comprising: Memory; computer readable instructions; as well as A processor circuit module, the processor circuit module is used to execute the computer readable instructions to: Determining a hybrid sparsity pattern of the selected layer of the artificial intelligence AI-based model, the hybrid sparsity pattern having a sparsity ratio and a block pattern for the selected layer; In response to the sparsity ratio being higher than a threshold, reducing the sparsity ratio of the selected layer; as well as In response to the sparsity ratio being lower than the threshold, a block mode of the selected layer is adjusted, the block mode of the selected layer corresponding to the accuracy ratio.

2. The device according to claim 1, wherein: The processor circuit module is used to reduce the sparsity ratio by reducing the number of zero weights for the selected layer.

3. The device according to claim 1, wherein: The processor circuit module is used to: in response to n modes of m of the block patterns of the selected layer being greater than a threshold corresponding to a hyperparameter, adjust the block pattern by reducing the number of elements per block in the selected layer.

4. The device according to claim 3, wherein: The processor circuit module is configured to: in response to a block size corresponding to the block mode being smaller than a threshold size, adjust the block mode to the n-of-m mode.

5. The device according to claim 1, wherein: The processor circuit module is used to determine the accuracy ratio based on a comparison of the output of the AI-based model with the labeled output of training data applied to the AI-based model, the output of the AI-based model being based on the training data.

6. The device according to claim 1, wherein: The processor circuit module is used to estimate the performance of the AI-based model based on a comparison of sparse operations and dense operations corresponding to the AI-based model.

7. The device according to claim 1, wherein: The processor circuit module is used to generate a kernel for each sparse weight of the selected layer of the AI-based model.

8. The device according to claim 7, wherein: The processor circuit module is used for: Dynamically sparsely encoding the AI-based model by selecting groupings of data in the AI-based model based on a computation to load ratio; as well as The kernel for each sparse weight is generated based on a dynamically encoded AI-based model.

9. A non-transitory computer-readable medium comprising instructions that, when executed, cause one or more processors to at least: Determining a hybrid sparsity pattern of the selected layer of the artificial intelligence AI-based model, the hybrid sparsity pattern having a sparsity ratio and a block pattern for the selected layer; In response to the sparsity ratio being above a threshold, reducing the sparsity ratio of the selected layer; and In response to the sparsity ratio being lower than the threshold, a block mode of the selected layer is adjusted, the block mode of the selected layer corresponding to the accuracy ratio.

10. The computer-readable medium of claim 9, wherein: The instructions cause the one or more processors to reduce the sparsity ratio by reducing a number of zero weights for the selected layer.

11. The computer-readable medium of claim 9, wherein: The instructions cause the one or more processors to adjust the block pattern by reducing the number of elements per block in the selected layer in response to n patterns of m of the block patterns of the selected layer being greater than a threshold corresponding to a hyperparameter.

12. The computer-readable medium of claim 11, wherein: The instructions cause the one or more processors to adjust the block mode to the n-of-m mode in response to a block size corresponding to the block mode being less than a threshold size.

13. The computer-readable medium of claim 9, wherein: The instructions cause the one or more processors to determine the accuracy ratio based on a comparison of an output of a selected layer with labeled outputs of training data applied to the AI-based model, the output of the selected layer being based on the training data.

14. The computer-readable medium of claim 9, wherein: The instructions cause the one or more processors to estimate performance of the AI-based model based on a comparison of sparse operations and dense operations corresponding to the AI-based model.

15. The computer-readable medium of claim 9, wherein: The instructions cause the one or more processors to generate a kernel for each sparse weight of the selected layer of the AI-based model.

16. The computer-readable medium of claim 15, wherein: The instructions cause the one or more processors to: dynamically sparsely encoding the AI-based model by selecting groupings of data in the AI-based model based on a computation to load ratio; as well as The kernel for each sparse weight is generated based on a dynamically encoded AI-based model.

17. A method comprising: Determining, by executing instructions with one or more processors, a hybrid sparsity pattern for a layer of an artificial intelligence (AI) based model, the hybrid sparsity pattern having a sparsity ratio and a block pattern for the layer; If the first sparsity ratio of the first layer is above a threshold, reducing the first sparsity ratio of the first layer by executing instructions with the one or more processors; as well as If the second sparsity ratio of the second layer is below the threshold, a block mode of the second layer is adjusted by executing instructions with the one or more processors, the block mode of the selected layer corresponding to the accuracy ratio.

18. The method according to claim 17, wherein: Reducing the first sparsity ratio includes reducing a number of zero weights for the first layer.

19. The method according to claim 17, wherein: Adjusting the block pattern includes: in response to an n-of-m pattern of the block pattern of the second layer being greater than a threshold corresponding to a hyperparameter, reducing the number of elements per block in the first layer.

20. The method according to claim 19, wherein: Adjusting the block mode includes: in response to a block size corresponding to the block mode being smaller than a threshold size, adjusting the block mode to the n-of-m mode.

21. The method of claim 17, further comprising determining the accuracy ratio based on a comparison of an output of the AI-based model with labeled outputs of training data applied to the AI-based model, the output of the AI-based model being based on the training data.

22. The method of claim 17, further comprising determining the accuracy ratio based on a comparison of an output of the AI-based model with labeled outputs of training data applied to the AI-based model, the output of the AI-based model being based on the training data.

23. The method of claim 17, further comprising estimating performance of the AI-based model based on a comparison of sparse operations and dense operations corresponding to the AI-based model.

24. The method of claim 17, further comprising generating a kernel for each sparse weight of at least one of the first layer or the second layer of the AI-based model.

25. The method according to claim 24, further comprising: dynamically sparsely encoding the AI-based model by selecting groupings of data in the AI-based model based on a computation to load ratio; as well as The kernel for each sparse weight is generated based on a dynamically encoded AI-based model.

Images