Compression of deep learning models

Abstract

The present invention provides an apparatus and method for efficiently and effectively compressing deep learning models. [Solution] A computer implementation method for an AI system including a deep learning model, comprising: determining a group of two or more consecutive layers from a plurality of layers defining the weight matrix of the model; applying singular value decomposition (SVD) to the weights of one layer in the group for each identified group of two or more consecutive layers; truncating at least a portion of the singular values ​​obtained from the application of SVD for each layer to which SVD has been applied, based on at least one truncation criterion; selecting multiple output channels of the deep learning model to be removed for each identified group of two or more consecutive layers, based on at least one weight importance criterion; and removing one or more input channels and / or one or more output channels for each identified group of two or more consecutive layers in order to maintain the shared dimensionality within the group based on the selected multiple output channels.

Description

[Technical Field]

[0001] The present disclosure relates to the field of artificial intelligence (AI) systems. More specifically, it relates to methods, apparatuses, and systems for compressing AI systems in the form of deep learning models. [Background Art]

[0002] Deep learning models such as large language models (LLMs) are becoming increasingly complex and resource-intensive. Larger deep learning models incur more complex handling and execution. As a result, there is an urgent need for methods to efficiently compress these models without sacrificing performance or accuracy.

[0003] Structured pruning is a known technique for compressing deep learning models. It works by identifying and removing redundant or less important portions of a model's weight matrix. This compression reduces the size and complexity of the model without significantly impacting performance. This is key for enabling the model to produce good results with lower resource usage.

[0004] While structured pruning is effective, it can be computationally expensive and time-consuming, particularly for large models such as LLMs and for pruning that involves multiple iterations of pruning and restoration processes. This is because the computational cost of pruning typically increases exponentially in proportion to the number of parameters of the model. Conventional pruning methods often rely on heuristic approaches that do not always find an optimal solution. [Summary of the Invention] [Problem to be Solved by the Invention]

[0005] Efficient and effective methods for compressing deep learning models with learnable parameters are useful when running such deep learning models and / or when running larger models that can be effectively run after their complexity and size have been reduced by the compression process. [Means for solving the problem]

[0006] explanation The first aspect relates to a method. This method, with respect to an artificial intelligence (AI) system, includes determining a group of two or more consecutive layers from among multiple layers of a deep learning model included in the AI ​​system. Each determined group is a group of two or more consecutive layers that share a common dimension.

[0007] Furthermore, this method involves applying singular value decomposition (SVD) to the weights of one of the layers within each group of two or more consecutive layers that have been determined.

[0008] This method further includes, for each layer to which SVD is applied, truncating at least a portion of the singular values ​​obtained from the application of SVD based on at least one truncation criterion.

[0009] This method also involves selecting multiple output channels of the deep learning model to be removed for each group of two or more consecutive layers determined, based on at least one weight importance criterion. The ratio of the number of output channels in the selected multiple output channels to the total number of output channels in each group is less than or equal to the ratio of the number of truncated singular values ​​in each group to the number of truncated singular values ​​obtained for each group.

[0010] Furthermore, this method includes, for each group of two or more determined consecutive layers, removing one or more input channels and / or one or more output channels based on a selection of multiple output channels, in order to maintain the shared dimensional size within the group.

[0011] This method enables effective pruning of AI systems by removing irrelevant, and sometimes the least relevant, data. In other words, it compresses the AI ​​system by removing data that has little impact on the system's output or output set.

[0012] The resulting AI systems achieve reductions of at least 25% in one or more of the following areas: storage, memory, and / or processing power. In some cases, reductions of 50% or more are possible, and in others, reductions of 50% to 85% are achieved. Furthermore, in certain cases, the reduction in processing power allows the AI ​​system to run on one or more central processing units or their cores with execution times equivalent to those of a graphics processing unit running an uncompressed AI system, thus reducing the need for a GPU.

[0013] Reducing storage, memory, and / or processing power can sometimes improve the throughput of an AI system. In some cases, an increase of approximately 50% or more and / or approximately 100% or more of the original throughput—that is, the number of samples per unit time in the prefill phase and / or the number of tokens per unit time in the decryption phase of the AI ​​system—can be achieved, nearly doubling that of an uncompressed AI system. In terms of energy, a compressed AI system may consume twice as much energy as an uncompressed system during the load phase, but it can be more than 25% (e.g., more than 40%) more energy efficient than an uncompressed system during the prefill and / or decryption phases. This significantly reduces energy consumption as the processing load of the AI ​​system increases. In some cases, the reductions in any of these examples result in a decrease in the accuracy of the AI ​​system (i.e., the accuracy of providing the output that is considered correct among the set of possible outputs) of less than 15%, and in some cases less than 10%, and / or less than 5% compared to the accuracy of the uncompressed AI system.

[0014] Therefore, the time it takes to run the AI ​​system is shorter than the time it takes to run the AI ​​system before compression, and given memory and storage limitations, the compressed AI system enables more complex deep learning models in the sense that it can place more layers and / or more complex layers that occupy the same or less memory and storage as a simpler uncompressed AI system.

[0015] This method reduces the complexity of pruning compared to existing solutions, particularly in terms of how the pruning process is performed, the given compression ratio, and, in some cases, the number of iterations required to achieve the optimal compression ratio. This reduces the time and cost of pruning, and these benefits become more pronounced as the scale of deep learning models and AI systems increases.

[0016] Deep learning models often have billions of parameters, many of which have very little relevance to the model's output. Examples include numerous attention heads that capture similar patterns, multilayer perceptron neurons with low firing frequencies or very small contributions, and weight matrices containing small amplitude values. The method described herein allows for pruning of some of these less relevant parameters.

[0017] This pruning is performed on deep learning models with learnable parameters, making it flexible and suitable for a wide range of AI systems, such as neural architectures and / or large-scale language models (LLMs).

[0018] SVD estimates the compression ratio achievable by each layer, preferably in an automated manner, without causing information loss that would impair the output accuracy of the AI ​​system, i.e., without causing the loss of information that is important for the proper operation of the AI ​​system.

[0019] Identify input or output channels within the weight matrix and rank less important channels to achieve the estimated compression ratio. This allows for a reduction or minimization of the number of learnable parameters while maintaining the AI ​​system's behavior after compression. Simultaneously, the cost of the pruning process in terms of computational resources and time is also reduced or minimized.

[0020] The layers in each group of two or more consecutive layers may be selected to determine the compression ratio limit of the compression that may apply to that group.

[0021] SVD decomposes each weight matrix and provides singular values ​​representing the significance or importance of different dimensions in each matrix. The larger the singular value, the more important the corresponding dimension is to the matrix structure. Subsequent truncation removes singular values ​​that are not the largest singular values. In some cases, the smallest singular value, or a portion of the smallest singular values, may also be removed. The percentage of singular values ​​removed represents the maximum percentage of channels that can be removed from the original weight matrix for compression.

[0022] The channels to be removed are primarily those containing redundant or noisy data, and for this purpose, channels are selected according to the setting of pruning limits based on the removed singular values. Channel removal is performed based on at least one weight importance criterion, such as redundant channel or noise channel search. Input and / or output channels may be removed based on such weight importance criteria to compress the deep learning model. In some embodiments, each group consisting of two or more consecutive layers contains only two consecutive layers.

[0023] In some embodiments, the determination step includes tracking computational dependencies between consecutive layers of multiple layers and determining consecutive layers that share a common dimension.

[0024] The number of output channels of one layer (e.g., the first layer) in a group of two or more consecutive layers is matched to the number of input channels of another layer (e.g., the second layer) within the group, whereby the layers of the group have a common dimension. Layers of the model can be analyzed as part of a method of determining computational dependencies between layers and establishing common dimensions shared between different layers.

[0025] In some embodiments, in the applying step, the layer of the group to which SVD is applied is the layer closest to the input of the deep learning model.

[0026] It has been found that determining the limit of compression rate using the layer closest to the input can improve the compression rate of a deep learning model with little or no influence on the output provided by the model.

[0027] In some embodiments, the applying step comprises converting a group of one layer or two or more consecutive layers into respective second-order tensors before applying SVD to the weights of one layer of the group.

[0028] SVD can be applied to any weight matrix that is a second-order tensor. Therefore, matrices of different sizes, such as higher-order tensors, for example, but not limited to, higher-order tensors such as convolution height, can first be unfolded or reshaped into a second-order tensor.

[0029] In some embodiments, the at least one truncation criterion comprises truncating singular values having values equal to or less than an absolute threshold.

[0030] In some embodiments, the at least one truncation criterion comprises truncating singular values that are equal to or less than the product of a relative threshold and the value of the maximum singular value.

[0031] Truncation of singular values is performed simply by comparing singular values with a threshold, and thus is simple and cost-effective. The threshold may be an absolute threshold or a relative threshold derived from the value of the maximum singular value.

[0032] In some embodiments, at least one truncation criterion includes truncating singular values ​​that are less than or equal to a variable threshold that depends on the value obtained by dividing the truncated singular value by the total value of all singular values.

[0033] The variable threshold is set such that, for example, truncation follows the following relationship: the sum of the squares of the truncated singular values ​​divided by the sum of the squares of all singular values. Other relationships are similarly possible, and it is not necessarily required to be the sum of squares.

number

[0034] In some embodiments, at least one weight importance criterion includes performing redundant channel search as follows: calculating the cosine similarity between all channel weights to provide a similarity matrix; selecting the channel weight pair with the highest cosine similarity; removing the channel weight of the selected channel weight pair that has fewer similarity candidates; removing the rows and columns of the similarity matrix corresponding to the removed channel weight; and repeating the selection and removal steps until the highest cosine similarity present in the similarity matrix is ​​less than or equal to a similarity threshold.

[0035] In some embodiments, at least one weight importance criterion includes performing a noise channel search. Specifically, a score for each channel weight is calculated by flattening all channel weights and calculating the sum of their squares. Then, all or some of the channel weights whose calculated scores fall below a score threshold are removed.

[0036] Redundant channel search and noisy channel search have been shown to appropriately select channels to remove from deep learning models for effective compression without impairing the model's performance.

[0037] In some embodiments, at least one weight importance criterion includes performing a redundant channel search before performing a noise channel search.

[0038] In some embodiments, the layers include convolutional layers. The steps to be applied include transforming the convolutional layers into their respective quadratic tensors and fusing the filter dimension of the convolutional layers with the smallest available channel dimension.

[0039] By reshaping the tensor in the convolutional layer into a matrix that is as close to a square shape as possible, the performance of SVD and subsequent truncation is improved.

[0040] In some embodiments, the layers include fully connected layers, each processed such that its number of input channels is equal to the input dimension, its number of output channels is equal to the output dimension, and the size of the input feature map is units.

[0041] Fully connected layers can be considered a special form of convolution. By processing them so that the number of input channels is equal to the input dimension, the number of output channels is equal to the output dimension, and the size of the input feature map is 1, similar to the kernel size and stride, fully connected layers can be pruned in the same way as convolutional layers. This is because the weight matrix is ​​already two-dimensional, making it suitable for SVD.

[0042] In some embodiments, the layers include an attention layer, which is processed such that if one or more input channels and / or one or more output channels are removed, a complete attention head with the corresponding one or more input channels and / or one or more output channels is removed.

[0043] The attention layer of the transformer model has four matrix multiplication operations that can be considered as fully connected layers: calculation of the key layer (K), query layer (Q), and value layer (V), and Attention(Q,K,V)=softmax(QK T This is the output fully connected layer calculated according to V. These four layers are pruned together, and instead of removing individual channels, the entire attention head is removed. Preferably, layers with a high proportion of less important channels are preferred.

[0044] The query matrix may be selected as the layer to which SVD and truncation are applied.

[0045] In some embodiments, in the decision-making step, the AI ​​system is a trained AI system.

[0046] Before compression, the AI ​​system may have already been trained using the training input dataset. This means that improved relationships already exist within the deep learning model, allowing for more accurate identification of which channels contain redundant or irrelevant data, thus enabling more effective compression.

[0047] In some embodiments, this method also includes, after the removal step, inputting the input vector to an AI system and obtaining at least one output vector from the AI ​​system.

[0048] In some embodiments, the method further includes deriving at least one instruction based on the obtained at least one output vector and applying at least one instruction to at least one target device.

[0049] In some embodiments, this method includes, after the removal step, using at least one derived instruction on a process-related device, system, or controller. This device, system, or controller is, for example, a process-related device, system, or controller.

[0050] In some embodiments, the input dataset used in the AI ​​system includes one or more measurements from at least one sensor and / or at least one computing device or device related to the process.

[0051] In some embodiments, this method also includes repairing a training input dataset (e.g., a given training input dataset) for training an AI system.

[0052] Dataset repair can improve the accuracy of compressed AI systems. In this sense, profiling the layers of a deep learning model may eliminate the need to repeat the repair step multiple times. For example, in some embodiments, repair may be performed only once or twice.

[0053] The restoration is carried out according to a predetermined restoration procedure. In some embodiments, the predetermined restoration procedure includes conventional restoration and / or knowledge distillation.

[0054] Traditional restoration methods involve fine-tuning the dataset used to train the original deep learning model (the deep learning model before compression) using supervised or unsupervised methods.

[0055] Knowledge distillation uses the original deep learning model as the training model and utilizes its output in one or more loss functions. In particular, the final output of the training model is provided as a new label. The distance between the student model and the training model's intermediate outputs (e.g., logits) may be reduced or minimized using different loss functions, such as the Kullback-Leibler divergence.

[0056] In some embodiments, this method also includes training an AI system using a training input dataset. In some embodiments, the training input dataset is a restored training input dataset. In some embodiments, the training step is performed after the removal step.

[0057] After compression, the AI ​​system may be trained for the first time or again (for example, a compressed AI system that was already trained before compression).

[0058] In some embodiments, the AI ​​system is a large-scale language model.

[0059] The second aspect relates to an apparatus or system, which comprises means for carrying out the method described in the first aspect.

[0060] The method is configured to determine groups of two or more consecutive layers within a deep learning model, for example, with respect to an artificial intelligence (AI) system that includes multiple layers defining the weight matrix of the model, where each group of two or more consecutive layers shares a common dimension.

[0061] Furthermore, this method is configured to apply singular value decomposition (SVD) to the weights of one layer within each group of two or more consecutive layers that have been determined.

[0062] Furthermore, this method is configured to, for example, truncate at least a portion of the singular values ​​obtained from the application of SVD for each layer to which SVD is applied, based on at least one truncation criterion.

[0063] This method is also configured to select multiple output channels of a deep learning model for elimination based on at least one weight importance criterion, for example, for each group of two or more determined consecutive layers. The ratio of the number of output channels in the multiple output channels to the total number of output channels in each group is less than or equal to the ratio of the number of truncated singular values ​​in each group to the number of truncated singular values ​​obtained for each group.

[0064] This means is configured, for example, to remove one or more input channels and / or one or more output channels for each determined group of two or more consecutive layers, based on a selection of multiple output channels, such that the shared dimension size within the group is maintained.

[0065] This means may include, in non-limiting examples, at least one processor and at least one memory module. This means may further, or instead, be a plurality of devices. Each device has at least one processor. In this case, at least one memory module may be shared by some or all of the devices, and / or some of the devices have at least one memory module each.

[0066] In some embodiments, the apparatus or system includes the same or additional means for performing the steps according to the embodiment of the first aspect.

[0067] A third aspect relates to a computer program that, when the program is executed by at least one processing unit or system, includes instructions causing at least one processing unit or system to perform the method described in the first aspect.

[0068] A fourth aspect relates to a computer-readable non-temporary storage medium that, when executed by at least one processing unit or system, includes instructions causing at least one processing unit or system to perform the method described in the first aspect.

[0069] The fifth aspect relates to a data carrier signal for transmitting the computer program described in the fourth aspect.

[0070] To complement the explanation and enhance understanding of the disclosure, a series of drawings are provided. These drawings constitute part of the specification and illustrate embodiments of the disclosure. They should not be interpreted as limiting the scope of the disclosure, but rather as examples of how the disclosure may be implemented. The drawings consist of the following figures. [Brief explanation of the drawing]

[0071] [Figure 1] Figure 1 shows an apparatus or system according to an embodiment. [Figure 2]Figure 2 shows the method according to an embodiment. [Figure 3] Figures 3 and 4 show a comparison of pruning results using the method according to the embodiment and pruning results using an existing solution. [Figure 4] Figures 3 and 4 show a comparison of pruning results using the method according to the embodiment and pruning results using an existing solution. [Modes for carrying out the invention]

[0072] Detailed explanation Figure 1 shows an apparatus or system 100 according to an embodiment.

[0073] The device or system 100 includes one or more processors 102. The device or system 100 also includes at least one memory module 104 for storing data such as an AI system or a deep learning model. Furthermore, at least one memory module 104 may store a computer program in the form of instructions that perform the method according to this disclosure at runtime.

[0074] In some embodiments, the device or system 100 may also include a communication module 106 configured to transmit and / or receive data to the computing device or system in wired and / or wireless manner. For example, the device or system 100 may transmit operation commands to and / or receive data from a controller that configures and / or controls the operation of the device or system.

[0075] In some embodiments, one or more processors 102 constitute, or are part of, at least one field-programmable gate array (i.e., FPGA), the at least one FPGA storing instructions and / or executing methods based on this disclosure. In some embodiments, the at least one FPGA is configured to run a quantum-inspired tensor network, thereby increasing the processing speed at which the at least one FPGA runs AI systems and associated methods.

[0076] Figure 2 shows a flowchart illustrating method 200 according to an embodiment. Method 200 enables the compression of a deep learning model, or an AI system that provides a deep learning model.

[0077] Method 200 includes, for example, the step of determining (202) a group of two or more consecutive layers from among multiple layers of a deep learning model, where at least one computing device (e.g., device) has multiple layers and defines the model's weight matrix. The deep learning model is part of an AI system. The group of two or more consecutive layers determined (202) share a common dimension.

[0078] Deep learning models, and by extension AI systems, are configured to provide at least one output based on an input dataset. They learn from the data to generate outputs such as predictions or decisions. Deep learning models and AI systems are involved in one of the following processes in several examples. These, along with other processes that may work with the disclosed deep learning models and AI systems, are some of the examples included in the scope of this disclosure.

[0079] 1. Manufacturing: Optimization of predictive maintenance The AI ​​system provides predictive maintenance models based on real-time sensor data from machines (e.g., vibration, temperature, etc.), or the output of such models (e.g., if the AI ​​system is such a model), in a format that prevents unexpected failures and reduces downtime.

[0080] 2. Energy sector: Demand forecast adjustments The AI ​​system can, by incorporating one or more of the following into its input dataset—weather changes, fluctuations in industrial consumption, increased energy generation by consumers, and expected high or low demand for electricity (e.g., a village partially uninhabited on a weekend, a social event, etc.)—preferably provide output in the form of one or more energy demand forecasts and / or changes in power source operation (e.g., curtailment of wind turbines or photovoltaic panels, increased production of wind turbines or photovoltaic panels, preparation of battery cells to provide stored energy, etc.) in real time. This optimizes energy flow and reduces reliance on non-renewable resources during peak times.

[0081] 3. Healthcare: Personalized Medicine AI systems can provide output in the form of treatment plans based on a patient's response to treatment. Measurements that assess a patient's health status can be part of the input dataset. For example, if cancer treatment is effective for a patient, the AI ​​system can improve clinical outcomes and personalize care by dynamically adjusting dosages or exploring new treatments.

[0082] 4. Transportation and Logistics: Dynamic Route Optimization A dynamic route optimization AI system can provide real-time output in the form of transportation routes based on traffic conditions, weather, geolocation data, and / or road restrictions, reducing delivery times, lowering costs, and reducing carbon emissions, one or more of which are part of the input dataset.

[0083] 5. Agriculture: Optimization of resource use The AI ​​system can provide irrigation and fertilization instructions and recommendations based on real-time soil and weather data. This minimizes water and fertilizer use while improving crop yields.

[0084] 6. Retail: Real-time price optimization AI systems can provide output in the form of pricing adjusted based on current demand, competitive landscape, and consumer behavior. This allows retailers to maximize profit margins, reduce inventory waste, and improve customer satisfaction.

[0085] 7. Cybersecurity: Adapting Threat Detection AI systems can present threat detection models based on new vulnerabilities and cyberattacks. This improves responses to emerging threats and enhances defenses against security breaches.

[0086] 8. Image Processing AI systems can provide output results, such as object detection and recognition within images. This makes it possible to accurately and efficiently determine the presence of objects in an image. For example, they can process satellite imagery to determine whether objects are present and, if so, identify the type of object.

[0087] Method 200 also includes step 204, in which at least one computing device applies SVD to the weights of one layer determined for each determined group (202).

[0088] The applied (204) SVD decomposes each weight matrix into three components in the following way: A = UΣV T Here, A is the weight matrix to be decomposed, and U∈R m×m and V T ∈R n×n Σ∈R is an orthogonal matrix. m×n is a diagonal matrix containing singular values. SVD is any weight matrix that is a quadratic tensor, i.e., A∈R m×n This is applicable (204). Therefore, if the weight matrix is ​​a higher-order tensor, it may first be converted to a second-order tensor in order to apply (204) SVD.

[0089] Method 200 further includes a step 206 in which at least one computing device truncates at least some of the singular values ​​provided by the application of SVD (204) for each layer to which SVD has been applied (204). The truncation 206 is performed based on at least one truncation criterion, such as an absolute threshold, a relative threshold, or a cumulatively truncated or discarded weight threshold.

[0090] Method 200 also includes step 208 in which at least one computing device selects multiple output channels for each determined (202) group of deep learning models to be removed. This selection 208 is based on at least one weight importance criterion. The ratio of the number of output channels in the selected (208) multiple output channels to the total number of output channels in each group is less than or equal to the ratio of the number of truncated (206) singular values ​​in each group to the number of truncated singular values ​​obtained for each group.

[0091] Method 200 further includes step 210 in which at least one computing device removes one or more input channels and / or one or more output channels for each determined (202) group, based on a selection (208) of multiple output channels, such that the shared dimension size within the group is maintained.

[0092] Compressed deep learning models, and consequently AI systems, reduce the memory footprint, inference latency, energy consumption, and bandwidth requirements during deployment. This allows AI systems to reduce latency in real-time perception and achieve faster processing with less memory space. As a result, AI systems can be adapted to hardware storage constraints, and even communication efficiency improves as more data is available. As energy consumption decreases, running AI systems becomes more cost-effective (e.g., 25% or more more cost-effective than the uncompressed version, 50% or more cost-effective). The battery life of the devices running the AI ​​systems is also extended.

[0093] In some embodiments, the application of SVD (204) includes estimating or determining the compression ratio for each layer of the deep learning model. In some embodiments, the estimation or determination of the compression ratio includes obtaining at least one threshold based on the correlation of the obtained singular values.

[0094] Therefore, in some embodiments, this estimation or determination may lead to adaptive layer-by-layer pruning, where each layer determines its rank (e.g., the optimal rank).

[0095] The use of thresholds helps remove redundancy while maintaining the beneficial structure of deep learning models, enabling an automatic balance between accuracy and compression ratio in AI systems. Unlike unstructured pruning techniques, fully structured compression tends to result in faster inference. Furthermore, because thresholds provide global control over pruning, compression can be achieved without manually searching for hyperparameters.

[0096] Correlation metrics can leverage the geometric properties of the weight space, but standard absolute-value-based methods cannot detect redundancy between singular vectors.

[0097] In some embodiments, Method 200 also includes the step of storing the deep learning model in a storage location where at least one computing device is connected. Preferred storage locations are those for uncompressed deep learning models, such as at least one memory module of at least one computing device, or at least one memory module of a cloud server connected to at least one computing device. In some cases, a compressed deep learning model may replace an uncompressed deep learning model.

[0098] Figure 3 shows a chart comparing, in percentage terms, the output accuracy of a deep learning model pruned using the method based on the embodiment and a model pruned using an existing solution.

[0099] The model pruned using this method, indicated by the black bars, has the following characteristics: it achieves a 60% compression ratio, uses approximately 20 billion training tokens, and does not employ supervised fine-tuning (SFT) or direct preference optimization (DPO).

[0100] One of the existing solutions used for comparison is Llama3.2-3B (3.2B), developed by Meta, indicated by a diagonal bar. Its characteristics are as follows: 60% compression is performed, 9 trillion training tokens are used (i.e., 450 times the number of training tokens used in pruning with this method), and both SFT and DPO are performed. Another existing solution used for comparison is Llama3-Minitron-4B (4.5B), developed by Nvidia. This is indicated by a dot bar and has the following characteristics: 45% compression is performed, 96 billion training tokens are used (i.e., approximately 5 times the number of training tokens used in pruning with this method), and neither SFT nor DPO is performed.

[0101] The comparison will be made using different datasets, specifically MMLU (Massive Multitask Language). This is being done for Understanding, BoolQ, HellaSwag, and GSM8K.

[0102] As can be seen from the figure, the deep learning model pruned using this method achieves accuracy equivalent to the comparison solution. In particular, the accuracy degradation compared to the highest accuracy level is only about 5-10%. Furthermore, in the case of the BoolQ dataset, the accuracy level of the model pruned using this method surpasses that of the model pruned using the existing solution.

[0103] Figure 4 is another chart showing the accuracy (percentage) of the output of a deep learning model pruned using the method based on the embodiment and using an existing method.

[0104] In addition to the characteristics of the method described in relation to Figure 3, the pruning of this method includes repair on publicly available data, whereas the pruning of existing methods includes repair on privately available data.

[0105] The achieved accuracy results are even better than in Figure 3, with less accuracy degradation on the MMLU dataset. Furthermore, on the GSM8K dataset, the accuracy is significantly better than the comparison solution. Also, on the WinoGrande dataset, the achieved accuracy is very close to that of the comparison solution.

[0106] While this specification describes specific examples, it is evident that various changes and modifications can be made to these examples without departing from the broad spirit and scope of the disclosure. Therefore, the specification and drawings should be interpreted as illustrative, not restrictive. The accompanying drawings constitute part of this specification and are illustrative, not restrictive, examples of how the subject matter can be carried out. The illustrated embodiments are described in sufficient detail to enable those skilled in the art to carry out the teachings disclosed herein. Other embodiments are also available and can be derived by structural and logical substitutions and modifications without departing from the scope of this specification. Therefore, this detailed description should not be interpreted as restrictive, and the scope of the various embodiments is defined solely by the entire scope of the accompanying claims and their equivalent scopes.

[0107] Such examples relating to the subject matter of the invention are referred to herein, individually or collectively, merely for convenience, by the terms "embodiments" or "examples," and where multiple examples are disclosed, there is no intention to arbitrarily limit the scope of this application to a single example or concept. Therefore, while specific examples are illustrated and described herein, it should be understood that any configuration designed to achieve the same objective may be used in place of any particular example shown. This disclosure is intended to encompass all adaptations or variations of various examples. Combinations of the above examples, and other examples not specifically described herein, will become apparent to those skilled in the art by considering the above description.

[0108] Some of the subjects discussed herein may involve presenting operations on data stored as bits or binary digital signals in machine memory (e.g., computer memory) using algorithms or symbolic representations. Such algorithms and symbolic representations are examples of techniques used by those with ordinary skill in the field of data processing to communicate the nature of the work to others skilled in the art. In this specification, “algorithm” refers to a self-contained sequence of operations or similar processes that produce a desired result. In this context, algorithms and operations involve the physical manipulation of physical quantities. Typically, but not necessarily, such physical quantities may take the form of electrical, magnetic, or optical signals that can be stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. For primarily conventional reasons, it may be convenient to refer to such signals with terms such as “data,” “content,” “bit,” “value,” “element,” “symbol,” “character,” “term,” “number,” or “digit.” However, these terms are merely labels for convenience and must be associated with the appropriate physical quantities.

[0109] Unless otherwise specified, descriptions using terms such as “processing,” “calculating,” “calculating,” “determining,” “presenting,” and “displaying” in this specification may refer to the operation or process of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a suitable combination thereof), registers, or other machine components that receive, store, transmit, or display information. Furthermore, unless otherwise specified, the terms “a” and “an” in this specification are used to include one or more instances, in accordance with common usage in patent documents. In this specification, the conjunction “or” means non-exclusive “or” unless otherwise specified.

[0110] Where the context allows, words used in singular or plural form also include their plural or singular forms, respectively. The word "or" in relation to a list of two or more items encompasses all of the following interpretations: any item in the list, all items in the list, and any combination of items in the list.

[0111] Some examples may include a specific sequence of operations, which may be modified without departing from the scope of this disclosure. For example, some of the illustrated operations may be performed in parallel or in a different order without substantially affecting the functionality described in the examples. In other examples, different components of an exemplary apparatus or system implementing an exemplary method may perform functions substantially simultaneously or in a specific order.

[0112] In this specification, the term “processor” may refer to one or more circuits or virtual circuits (e.g., physical circuits emulated by logic running on an actual processor) that manipulate data values ​​according to control signals (e.g., commands, opcodes, machine code, control words, macro instructions, etc.) and generate corresponding output signals applied to operate a machine. Processors include, for example, central processing units (CPUs), reduced instruction set computing (RISC) processors, complex instruction set computing (CISC) processors, graphics processing units (GPUs), digital signal processing units (DSPs), tensor processing units (TPUs), neural processing units (NPUs), vision processing units (VPUs), machine learning accelerators, artificial intelligence accelerators, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), radio frequency integrated circuits (RFICs), neuromorphic processors, quantum processors, or any combination thereof. A processor may be a multi-core processor having two or more independent processors (sometimes called “cores”) that can execute instructions simultaneously. A multicore processor contains multiple arithmetic cores on a single integrated circuit die, each capable of executing program instructions independently and in parallel. Parallel processing in multicore processors is achieved through architectures such as superscalar, VLIW, vector processing, and SIMD, where each core can simultaneously execute separate instruction streams. Processors can be implemented as software emulations, virtual processors, or virtual circuits running on a physical processor. A virtual processor behaves like an independent processor but is implemented in software rather than hardware.

[0113] Various operations of the exemplary methods described herein may be performed, at least partially, by one or more processing units configured temporarily (e.g., by software) or permanently to perform the operations in question. Whether configured temporarily or permanently, such processors may constitute processor implementation modules / components that operate to perform one or more operations or functions. The modules / components referred to herein may, in some examples, include processor implementation modules / components.

[0114] Similarly, the methods described herein may be, at least partially, processor implementations. For example, at least some operations of a method may be performed by one or more processors or processor implementation modules / components. The execution of a particular operation may reside not only within a single machine but also distributed across one or more processors located on multiple machines. In some examples, the processors may be located in a single location (e.g., a home environment, an office environment, or a server farm), while in other examples, the processors may be distributed across multiple locations.

[0115] The embodiments are implemented in digital electronic circuits, computer hardware, firmware, software, or a combination thereof. The embodiments are implemented using computer programs embodied in computer program products, such as information carriers (e.g., data processing devices, e.g., programmable processors, computers, or machine-readable media for execution by or control of the operation of multiple computers).

[0116] Computer programs can be written in any form of programming language, including compiled and interpreted languages, and can be deployed in any form, either as standalone programs or as modules, subroutines, or other units suitable for use in computing environments. Computer programs can be deployed to run on a single computer, on multiple computers within a single site, or distributed across multiple sites interconnected by a communication network.

[0117] In this specification, “includes,” “composes,” and their derivatives (such as “inclusion,” “composition,” etc.) should not be understood in an exclusive sense (i.e., an interpretation that excludes the possibility of additional elements, steps, etc., being included in what is described or defined).

[0118] On the other hand, this disclosure is clearly not limited to the specific embodiments described herein, but also encompasses any modifications that a person skilled in the art could consider within the general scope of the claimed invention (for example, with respect to the selection of materials, dimensions, components, configurations, etc.).

Claims

1. It is a device, At least one processor, It comprises at least one memory module, The at least one processor cooperates with the at least one memory module, In an artificial intelligence (AI) system including a deep learning model having multiple layers that define the weight matrix of the model, the determination of two or more consecutive groups of layers among the multiple layers, wherein each of the two or more consecutive groups of layers shares a common dimension. For each group of two or more consecutive layers determined, apply singular value decomposition (SVD) to the weight of one layer in the group. For each layer to which SVD is applied, at least a portion of the singular values ​​obtained from the application of SVD is truncated based on at least one truncation criterion. For each group of two or more consecutive layers determined, select multiple output channels of the deep learning model to be removed based on at least one weight importance criterion, wherein the ratio of the number of output channels in the selected multiple output channels divided by the total number of output channels in each group is less than or equal to the ratio of the number of truncated singular values ​​in each group divided by the number of truncated singular values ​​obtained for each group, and For each group of two or more determined consecutive layers, remove one or more input channels or at least one of one or more output channels based on a selection of multiple output channels, such that the size of the shared dimension within the group is maintained. An apparatus configured to cause the aforementioned apparatus to perform the above-mentioned action.

2. The apparatus according to claim 1, wherein the determination of the group includes tracking computational dependencies between consecutive layers of a plurality of layers and determining consecutive layers that share a common dimension.

3. The apparatus according to claim 1, wherein when the SVD is applied, the layer closest to the input of the deep learning model is one of the layers in the group to which the SVD is applied.

4. The apparatus according to claim 1, wherein the application of the SVD includes converting one layer or a group of two or more consecutive layers into quadratic tensors before applying the SVD to the weights of one layer in the group.

5. In the apparatus according to claim 1, the at least one truncation criterion is: Truncation of singular values ​​that are below the absolute threshold, or Truncate singular values ​​that have a value less than or equal to the product of the relative threshold and the maximum singular value, or Truncation of singular values ​​that have a value less than or equal to a variable threshold that depends on the value obtained by dividing the truncated singular value by the value of all singular values. A device that includes one or more of the following.

6. In the apparatus according to claim 1, at least one weight importance criterion includes performing a search for redundant channels or a search for noisy channels, The search for redundant channels is performed by calculating the cosine similarity between all channel weights to provide a similarity matrix, selecting the pair of channel weights with the highest cosine similarity, removing the channel weight with fewer similarity candidates from the selected pair, removing the rows and columns of the similarity matrix corresponding to the removed channel weight, and repeating the selection and removal steps until the highest cosine similarity in the similarity matrix is ​​less than or equal to a similarity threshold. The apparatus performs the noise channel search by flattening all channel weights, calculating the sum of their squares to obtain a score for each channel weight, and removing all or some of the channel weights whose calculated scores fall below a score threshold.

7. The apparatus according to claim 6, wherein the at least one weight importance criterion includes searching for redundant channels and searching for noise channels, the searching for redundant channels precedes the searching for noise channels.

8. The apparatus according to claim 1, wherein the plurality of layers include convolutional layers, and the application of the SVD includes transforming the convolutional layers into their respective quadratic tensors such that the filter dimension of the convolutional layer is fused with the minimum available channel dimension.

9. The apparatus according to claim 1, wherein the plurality of layers include a fully connected layer, and the fully connected layer is processed such that the number of input channels is equal to the input dimension, the number of output channels is equal to the output dimension, and the size of the input feature map is a unit.

10. The apparatus according to claim 1, wherein the plurality of layers include an attention layer, and the attention layer is processed such that when at least one of one or more input channels or one or more output channels is removed, a complete attention head including at least one of each of the one or more input channels or one or more output channels is removed.

11. The apparatus according to claim 1, wherein the AI ​​system used to determine the group is a trained AI system.

12. In the apparatus according to claim 1, the at least one processor, together with the at least one memory module, provides the apparatus with To repair the training input dataset for training the aforementioned AI system, and To train an AI system using the aforementioned training input dataset, A device configured to perform a certain action.

13. The apparatus according to claim 1, wherein the application of SVD includes estimating the compression ratio of each layer of the deep learning model.

14. The apparatus according to claim 13, wherein the estimation of the compression ratio includes obtaining at least one threshold based on the correlation of the singular values ​​obtained from the application of the SVD.

15. A computer implementation method, The artificial intelligence (AI) system includes a deep learning model having multiple layers that define the weight matrix of the model, and the step of determining two or more consecutive groups of layers from the multiple layers, wherein each of the two or more consecutive groups of layers shares a common dimension. For each group of two or more consecutive layers determined, apply singular value decomposition (SVD) to the weights of one layer in the group. For each layer to which SVD is applied, the step of discarding at least a portion of the singular values ​​obtained from the application of SVD based on at least one discard criterion, A step of selecting, for each group of two or more determined consecutive layers, to remove multiple output channels of the deep learning model based on at least one weight importance criterion, wherein the ratio of the number of output channels in the selected multiple output channels divided by the total number of output channels in each group is less than or equal to the ratio of the number of truncated singular values ​​in each group divided by the number of truncated singular values ​​obtained for each group; For each group of two or more determined consecutive layers, the steps include removing one or more input channels or at least one of one or more output channels based on a selection of multiple output channels, such that the size of the shared dimension within the group is maintained; A method that includes [a certain feature].

16. A computer implementation method according to claim 16, wherein the determining step includes tracking computational dependencies between consecutive layers of a plurality of layers and determining consecutive layers that share a common dimension.

17. A computer implementation method according to claim 16, wherein in the step of applying, the layer closest to the input of the deep learning model is one of the layers in the group to which SVD is applied.

18. A computer implementation method according to claim 16, wherein the applying step includes transforming one layer or a group of two or more consecutive layers into quadratic tensors before applying the SVD to the weights of one layer in the group.

19. A computer implementation method according to claim 16, wherein the at least one truncation criterion is Truncation of singular values ​​that are below the absolute threshold, or Truncate singular values ​​that have a value less than or equal to the product of the relative threshold and the maximum singular value, or Truncation of singular values ​​that have a value less than or equal to a variable threshold that depends on the value obtained by dividing the truncated singular value by the value of all singular values. A method that includes one or more of the above.

20. A computer-readable non-temporary storage medium, when executed by at least one processing unit or system, includes instructions that cause the at least one processing unit or system to perform the following steps: The step of determining a group of two or more consecutive layers among the multiple layers of an artificial intelligence (AI) system that includes a deep learning model having multiple layers defining the weight matrix of the model, wherein each of the two or more consecutive layers shares a common dimension. For each group of two or more consecutive layers determined, apply singular value decomposition (SVD) to the weights of one layer in the group. For each layer to which SVD is applied, a step of discarding at least a portion of the singular values ​​obtained from the application of SVD based on at least one truncation criterion, A step of selecting, for each group of two or more consecutive layers determined, to remove multiple output channels of the deep learning model based on at least one weight importance criterion, wherein the ratio of the number of output channels in the selected multiple output channels divided by the total number of output channels in each group is less than or equal to the ratio of the number of truncated singular values ​​in each group divided by the number of truncated singular values ​​obtained for each group, and For each group of two or more determined consecutive layers, the step of removing one or more input channels or at least one of one or more output channels based on a selection of multiple output channels, such that the size of the shared dimension within the group is maintained.

Images