Deep learning model parameter self-adaptive pruning method and device, equipment and medium

Abstract

This application relates to a method, apparatus, device, and medium for adaptive parameter pruning of a deep learning model. The method includes: acquiring a pre-set training set of a deep learning model; pre-training the deep learning model based on the training set to determine the parameters of each layer in the deep learning model; capturing the low-rank structure of the parameters of each layer in the deep learning model based on a unified tensor network model, and decomposing the parameters into multiple ring factor tensors and a core factor tensor; constructing an adaptive parameter pruning model based on a sparse Bayesian learning algorithm according to the prior distribution results of the Gaussian distributions corresponding to the ring factor tensors and the core factor tensor, and the prior distribution results of the Gamma distribution corresponding to the noise level; and pruning irrelevant components of the parameters based on the adaptive parameter pruning model that has reached convergence to determine the corresponding compressed parameters. This application enables the deep learning model to maintain a low number of parameters while still maintaining high accuracy.

Description

Technical Field

[0001] This application relates to the field of data processing, and in particular to a method for adaptive pruning of deep learning model parameters, a corresponding device, electronic equipment, and computer-readable storage medium. Background Technology

[0002] Deep learning refers to machine learning techniques based on deep neural network algorithms. It primarily uses models with multiple nonlinear transformation structures to process and analyze data. These models are commonly referred to as deep learning models and have numerous applications in artificial intelligence, decision-making, and other scenarios, such as image recognition, speech recognition, data mining, natural language translation, and computer game playing. Typically, a model library can be built from one or more different deep learning models.

[0003] Currently, deep learning models have achieved significant performance results in many computer vision tasks, such as image classification, video recognition, and object detection. However, existing deep learning models are limited by their large number of parameters, which leads to high memory usage and computational time consumption.

[0004] Existing deep learning models in the current technology are limited by the large number of parameters, which leads to technical problems such as high memory usage and very time-consuming computation. The applicant has made corresponding explorations to solve this problem. Summary of the Invention

[0005] The purpose of this application is to solve the above-mentioned problems by providing a method for adaptive pruning of deep learning model parameters, a corresponding device, electronic device, and computer-readable storage medium.

[0006] To achieve the various objectives of this application, the following technical solution is adopted:

[0007] A method for adaptive pruning of deep learning model parameters, proposed to meet one of the purposes of this application, includes the following steps:

[0008] Obtain a training set for a pre-defined deep learning model, pre-train the deep learning model based on the training set, and determine the parameters of each layer in the deep learning model.

[0009] The low-rank structure of the parameters of each layer in the deep learning model is captured based on the unified tensor network model, and the parameters are decomposed into multiple ring factor tensors and a core factor tensor.

[0010] Based on the sparse Bayesian learning algorithm, each element in the ring factor tensor and the core factor tensor is distributed by a Gaussian prior distribution to determine the prior distribution results of the Gaussian distribution corresponding to the ring factor tensor and the core factor tensor. The noise level in the deep learning model is distributed by a Gamma prior distribution to determine the prior distribution results of the Gamma distribution corresponding to the noise level. Based on the prior distribution results of the Gaussian distribution corresponding to the ring factor tensor and the core factor tensor and the prior distribution results of the Gamma distribution corresponding to the noise level, a parameter adaptive pruning model is constructed.

[0011] The problem of minimizing the parameters is determined, and the minimization problem is iterated based on the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches a convergent state. Based on the convergent parameter adaptive pruning model, irrelevant components of the parameters are pruned to determine the corresponding compression parameters, thus completing the adaptive pruning of the deep learning model parameters.

[0012] Optionally, after the step of decomposing the parameters into multiple ring factor tensors and a core factor tensor, the method includes:

[0013] By pairwise concatenation of the multiple ring factor tensors and capturing the relationships between them based on the core factor tensor, the reconstructed low-rank parameter tensor can be determined. It is represented as:

[0014]

[0015] in, To reconstruct the low-rank parameter tensor, n∈[1,N] is the N+1 order ring factor tensor of the unified tensor network model. The core factor tensor for unifying tensor network models.

[0016] Optionally, before the step of performing a Gaussian prior distribution on each element of the ring factor tensor and the core factor tensor based on the sparse Bayesian learning algorithm, the method includes:

[0017] Determine the reconstructed low-rank parameter tensor corresponding to the parameters of the deep learning model, and determine its corresponding likelihood function based on the reconstructed low-rank parameter tensor. The reconstructed low-rank parameter tensor is... The likelihood function is: in, This represents the reconstruction of the low-rank parameter tensor, the... Let τ represent the likelihood function corresponding to the reconstructed low-rank parameter tensor, and let τ represent the noise level. Represents the reconstruction of low-rank parameter tensors Each element conforms to a mean of The variance is τ -1 The Gaussian distribution, the express The modulus-n expansion matrix i n -th vectors.

[0018] Optionally, the step of determining the prior distribution results of the Gaussian distributions corresponding to the cyclic factor tensor and the core factor tensor by performing a Gaussian distribution on each element of the cyclic factor tensor and the core factor tensor based on the sparse Bayesian learning algorithm includes:

[0019] For each of the aforementioned ring factor tensors, a prior distribution of a Gaussian distribution is applied to each of its corresponding elements to determine the prior distribution result of the Gaussian distribution corresponding to the ring factor tensor. The prior distribution result of the Gaussian distribution corresponding to the ring factor tensor is as follows:

[0020]

[0021] Among them, the Represents the cyclic factor tensor, Λ i,j =diag(λ i,j ), i∈[1,N-1],j∈[1,N],i <j,U n =diag(u n ), the U n Let represent the covariance matrix.

[0022] Optionally, the step of determining the prior distribution results of the Gaussian distributions corresponding to the cyclic factor tensor and the core factor tensor by performing a Gaussian distribution on each element of the cyclic factor tensor and the core factor tensor based on the sparse Bayesian learning algorithm includes:

[0023] For the core factor tensor, a prior distribution of Gaussian distribution is applied to each of its corresponding elements to determine the prior distribution result of the Gaussian distribution corresponding to the core factor tensor. The prior distribution result of the Gaussian distribution corresponding to the core factor tensor is expressed as follows: Here, β is a scale parameter.

[0024] Optionally, the step of determining the prior distribution result of the Gamma distribution corresponding to the noise level by taking the noise level in the deep learning model as a prior distribution of the Gamma distribution includes:

[0025] For the noise level τ, a prior distribution of the Gamma distribution is performed on the noise level τ to determine the prior distribution result of the Gamma distribution corresponding to the noise level τ. The prior distribution result of the Gamma distribution corresponding to the noise level τ is expressed as follows:

[0026] Optionally, the step of determining the minimization problem of the parameters and iterating the minimization problem based on the alternating direction multiplier algorithm includes:

[0027] The problem of minimizing the parameters is

[0028]

[0029] in,

[0030] A deep learning model parameter adaptive pruning device provided for another purpose of this application includes:

[0031] The parameter acquisition module is configured to acquire a training set of a preset deep learning model, pre-train the deep learning model based on the training set, and determine the parameters of each layer in the deep learning model.

[0032] The parameter decomposition module captures the low-rank structure of the parameters of each layer in the deep learning model based on the unified tensor network model, and decomposes the parameters into multiple ring factor tensors and a core factor tensor.

[0033] The pruning model determination module is configured to perform a Gaussian prior distribution on each element of the ring factor tensor and the core factor tensor based on the sparse Bayesian learning algorithm, determine the prior distribution results of the Gaussian distribution corresponding to the ring factor tensor and the core factor tensor, perform a Gamma prior distribution on the noise level in the deep learning model, determine the prior distribution results of the Gamma distribution corresponding to the noise level, and construct a parameter adaptive pruning model based on the prior distribution results of the Gaussian distribution corresponding to the ring factor tensor and the core factor tensor and the prior distribution results of the Gamma distribution corresponding to the noise level.

[0034] The parameter pruning module is configured to determine the minimization problem of the parameters, iterate the minimization problem based on the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches a convergent state, and prune irrelevant components of the parameters based on the convergent parameter adaptive pruning model to determine the corresponding compression parameters, thereby completing the adaptive pruning of the deep learning model parameters.

[0035] An electronic device provided for another purpose of this application includes a central processing unit and a memory, the central processing unit being configured to invoke and run a computer program stored in the memory to perform the steps of the deep learning model parameter adaptive pruning method described in this application.

[0036] A computer-readable storage medium is provided for another purpose of this application, which stores, in the form of computer-readable instructions, a computer program implemented according to the deep learning model parameter adaptive pruning method, which, when called by a computer, executes the steps included in the corresponding method.

[0037] Compared to existing technologies, this application addresses the technical problems of existing deep learning models being limited by a large number of parameters, leading to high memory usage and computational time consumption. This application uses a unified tensor network model to capture the low-rank structure of the parameters in each layer of the deep learning model, decomposing the parameters into multiple cyclic factor tensors and a core factor tensor. Based on a sparse Bayesian learning algorithm, each element in the cyclic factor tensor and the core factor tensor is distributed using a Gaussian prior distribution to determine the corresponding Gaussian prior distribution results for the cyclic factor tensor and the core factor tensor. Finally, the noise level in the deep learning model is distributed using a Gamma distribution. The prior distribution of the noise level is determined by identifying the prior distribution of the Gamma distribution. Based on the prior distributions of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor, and the prior distribution of the Gamma distribution corresponding to the noise level, a parameter adaptive pruning model is constructed. The parameter minimization problem is determined, and iteratively applied using the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches convergence. Based on the converged parameter adaptive pruning model, irrelevant components of the parameters are pruned to determine the corresponding compression parameters. This application includes, but is not limited to, the following beneficial effects:

[0038] First, this application introduces a novel tensor network model, which is a generalized version of all other tensor network models, and has more powerful representation capabilities and a low-rank structure.

[0039] Secondly, the novel tensor network model proposed in this application can adaptively learn tensor network structures within a sparse Bayesian learning framework, thus enabling it to learn corresponding tensor network representations for different real-world data.

[0040] Thirdly, this application provides an adaptive tensor network model, which compresses the parameters in the deep learning model, greatly improving the computational speed of data processing and reducing the memory required for the deep learning model. This allows the deep learning model to maintain a low number of parameters while still maintaining its high accuracy, thus avoiding affecting the accuracy of the deep learning model. Attached Figure Description

[0041] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:

[0042] Figure 1 This is a flowchart illustrating the adaptive pruning method for deep learning model parameters in an embodiment of this application.

[0043] Figure 2 This is a probabilistic graphical model of the Unified Tensor Network (UTN) model in the embodiments of this application;

[0044] Figure 3 This is a block diagram illustrating the principle of the adaptive pruning device for deep learning model parameters in the embodiments of this application.

[0045] Figure 4 This is a schematic diagram of the structure of the computer device in the embodiments of this application. Detailed Implementation

[0046] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0047] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this application means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless coupling. The term “and / or” as used herein includes all or any units and all combinations of one or more associated listed items.

[0048] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.

[0049] Those skilled in the art will understand that the terms "client," "terminal," and "terminal device" as used herein include both devices that receive wireless signals, devices that only possess wireless signal receiver capabilities without transmission capabilities, and devices with receiving and transmitting hardware, devices that have receiving and transmitting hardware capable of bidirectional communication over a bidirectional communication link. Such devices may include: cellular or other communication devices such as personal computers or tablets, having single-line displays, multi-line displays, or cellular or other communication devices without multi-line displays; PCS (Personal Communications Service) that can combine voice, data processing, fax, and / or data communication capabilities; PDA (Personal Digital Assistant) that may include a radio frequency receiver, pager, internet / intranet access, web browser, notepad, calendar, and / or GPS (Global Positioning System) receiver; and conventional laptops and / or handheld computers or other devices that have and / or include radio frequency receivers. As used herein, "client," "terminal," and "terminal device" can be portable, transportable, installed in a means of transportation (air, sea, and / or land), or suitable and / or configured to operate locally and / or in a distributed manner, operating in any other location on Earth and / or in space. "Client," "terminal," and "terminal device" as used herein can also be a communication terminal, an internet access terminal, or a music / video playback terminal, such as a PDA, a MID (Mobile Internet Device), and / or a mobile phone with music / video playback capabilities, or a smart TV, set-top box, etc.

[0050] The hardware referred to by the names "server," "client," and "service node" in this application is essentially an electronic device with the equivalent capabilities of a personal computer. It is a hardware device with the necessary components revealed by the von Neumann architecture, such as a central processing unit (including an arithmetic logic unit and a control unit), memory, input devices, and output devices. The computer program is stored in its memory, and the central processing unit loads the program stored in the secondary storage into the main memory to run it, execute the instructions in the program, and interact with the input and output devices to complete specific functions.

[0051] It should be noted that the concept of "server" used in this application can also be extended to the case of server clusters. Based on the network deployment principles understood by those skilled in the art, the servers should be logically divided. Physically, these servers can be independent of each other but accessible through interfaces, or they can be integrated into a single physical computer or a computer cluster. Those skilled in the art should understand this flexibility and should not use it to constrain the implementation of the network deployment method in this application.

[0052] Unless otherwise expressly specified, one or more technical features of this application may be deployed on a server and accessed by a client through remote invocation of the online service interface provided by the server, or they may be directly deployed and run on a client for access.

[0053] Unless otherwise specified, the neural network models referenced or potentially referenced in this application may be deployed on a remote server and invoked remotely on the client, or deployed on a client with the capability to invoke directly. In some embodiments, when running on the client, the corresponding intelligence may be acquired through transfer learning in order to reduce the requirements on the client's hardware resources and avoid excessive consumption of the client's hardware resources.

[0054] Unless otherwise specified, all data involved in this application may be stored remotely on a server or on a local terminal device, as long as it is suitable for use by the technical solution of this application.

[0055] Those skilled in the art will understand that although the various methods in this application are described based on the same concept and thus present commonality among them, they can be performed independently unless otherwise specified. Similarly, the various embodiments disclosed in this application are all based on the same inventive concept; therefore, concepts expressed in the same way, as well as concepts that are appropriately changed for convenience but are expressed differently, should be understood equivalently.

[0056] Unless otherwise expressly stated, the various embodiments disclosed in this application can be combined in a cross-cutting manner to flexibly construct new embodiments, as long as such combination does not depart from the inventive spirit of this application and can meet the needs of the prior art or solve a certain deficiency in the prior art. Those skilled in the art should be aware of such modifications.

[0057] Based on the above exemplary scenarios, please refer to Figure 1 as well as Figure 2 In one embodiment of the deep learning model parameter adaptive pruning method of this application, the following steps are included:

[0058] Step S10: Obtain the training set of the preset deep learning model, pre-train the deep learning model according to the training set, and determine the parameters of each layer in the deep learning model.

[0059] The deep learning model can be an image recognition model, an object detection model, etc. A pre-defined training set for the deep learning model is obtained. This training set can be any data, such as image data, hyperspectral data, etc. The training set is determined before training the deep learning model. Where, x n Let y represent the training sample. n The sample labels are used to pre-train the deep learning model based on the training set, thereby determining the parameters of each layer in the deep learning model.

[0060] Step S20: Capture the low-rank structure of the parameters of each layer in the deep learning model based on the unified tensor network model, and decompose the parameters into multiple ring factor tensors and a core factor tensor.

[0061] By capturing the low-rank structure of the parameters of each layer in the deep learning model using a unified tensor network model, and decomposing the parameters into multiple cyclic factor tensors and a core factor tensor, the unified tensor network model (UTN model) can be used to capture the parameters of each layer in the deep learning model. The low-rank structure of the Unified Tensor Network (UTN model) is achieved by using a single original N-order parameter. It is decomposed into N third-order ring factor tensors and one Nth-order core factor tensor.

[0062] Furthermore, the multiple ring factor tensors are connected pairwise, and the relationships between the ring factor tensors are captured based on the core factor tensor. Specifically, the ring factor tensors are connected pairwise, and the relationships between all ring factor tensors are captured through a core factor tensor to determine the reconstructed low-rank parameter tensor. It is represented as:

[0063]

[0064] in, To reconstruct the low-rank parameter tensor, n∈[1,N] is the N+1 order ring factor tensor of the unified tensor network model. The core factor tensor for unifying tensor network models.

[0065] For ease of explanation, we can use... This represents the Unified Tensor Network (UTN) model. Similarly, we define... It is a tensor The modulus-n expansion matrix is ​​obtained, and the others are similarly calculated.

[0066] Step S30: Based on the sparse Bayesian learning algorithm, perform a prior Gaussian distribution on each element of the ring factor tensor and the core factor tensor to determine the prior Gaussian distribution results corresponding to the ring factor tensor and the core factor tensor. Perform a prior Gamma distribution on the noise level in the deep learning model to determine the prior Gamma distribution results corresponding to the noise level. Construct a parameter adaptive pruning model based on the prior Gaussian distribution results corresponding to the ring factor tensor and the core factor tensor and the prior Gamma distribution results corresponding to the noise level.

[0067] For each of the aforementioned ring factor tensors, a prior distribution of a Gaussian distribution is applied to each of its corresponding elements to determine the prior distribution result of the Gaussian distribution corresponding to the ring factor tensor. The prior distribution result of the Gaussian distribution corresponding to the ring factor tensor is as follows:

[0068]

[0069] Among them, the Represents the cyclic factor tensor, Λ i,j =diag(λ i,j ), i∈[1,N-1],j∈[1,N],i <j,U n =diag(u n ), the U n Let represent the covariance matrix.

[0070] For the core factor tensor, a prior distribution of Gaussian distribution is applied to each of its corresponding elements to determine the prior distribution result of the Gaussian distribution corresponding to the core factor tensor. The prior distribution result of the Gaussian distribution corresponding to the core factor tensor is expressed as follows: Here, β is a scale parameter.

[0071] For the noise level τ, a prior distribution of the Gamma distribution is carried out for the noise level τ, and a prior distribution result of the Gamma distribution corresponding to the noise level τ is determined. The prior distribution result of the Gamma distribution corresponding to the noise level τ is expressed as

[0072] A parameter adaptive pruning model is constructed according to the prior distribution results of the Gaussian distribution corresponding to the loop factor tensor and the core factor tensor, and the prior distribution results of the Gamma distribution corresponding to the noise level. Specifically, the loop factor tensor and the core factor tensor of the unified tensor network model (UTN model) can be modeled based on the sparse Bayesian learning technique, as follows:

[0073] For a given parameter The parameter The likelihood function of Can be expressed as:

[0074]

[0075] Where τ represents the noise level, Represents a Gaussian distribution with mean μ and variance σ 2 Of

[0076] The above formula indicates that each element of the parameter w conforms to a Gaussian distribution with mean And variance τ -1 Of, where Represents The modulo-n expansion matrix of The i n -th vector.

[0077] For each loop factor tensor n ∈ [1, N], each of its elements is given a prior distribution of a Gaussian distribution:

[0078]

[0079] Where Λ i,j = diag(λ i,j ), i ∈ [1, N - 1], j ∈ [1, N], i < j and U n = diag(u n ) represent covariance matrices. The above formula indicates that each element of the parameter [[ID=**60**]]Conforms to a Gaussian distribution with mean 0 and variance Of. Since the prior distribution mean of the parameter Is 0, when we maximize the posterior probability, each element of the parameter Will also tend to 0, so we can obtain Sparse representation

[0080] For the core factor tensor A prior distribution of a Gaussian distribution is also placed on each of its elements:

[0081]

[0082] where β is a scale parameter. The above equation represents that each element of the parameter conforms to a Gaussian distribution with a mean of 0 and a variance of Since the prior distribution mean of the parameter is 0, when we maximize the posterior probability, each element of the parameter will also tend to 0, so we can obtain Sparse representation

[0083] For each λ i,j , i ∈ [1, N - 1], j ∈ [1, N], i < j, a prior of a Gamma distribution is placed on each of its elements:

[0084]

[0085] where Ga(x|a, b) represents a Gamma distribution with parameters a and b. Each λ i,j controls the rank R of the loop factor tensor and other loop factor tensors i,j , when λ i,j is larger, the covariance of the corresponding and will be smaller, meaning that more of its elements will tend to the mean 0. When the irrelevant components are trimmed, the value of R i,j will also decrease accordingly, thus achieving automatic trimming of the parameters.

[0086] For each u n , n ∈ [1, N], a prior of a Gamma distribution is placed on each of its elements:

[0087]

[0088] Similarly, each u n controls the rank L of the loop factor tensor and the core factor tensor n , when u n is larger, the covariance of the corresponding and will be smaller, meaning that more of its elements will tend to the mean 0. When the irrelevant components are trimmed, the value of L n will also decrease accordingly, thus achieving automatic trimming of the parameters.

[0089] Finally, the noise level τ is given a prior of a Gamma distribution:

[0090]

[0091] It can be seen that as τ increases, the parameter The covariance of the prior distribution will be smaller, meaning that the parameters... and The fit will be higher. All the above parameters can be learned automatically within the sparse Bayesian learning framework, without requiring us to manually adjust them.

[0092] Step S40: Determine the minimization problem of the parameters, iterate the minimization problem based on the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches a convergent state, prune the irrelevant components of the parameters based on the converged parameter adaptive pruning model to determine the corresponding compression parameters of the parameters, and complete the adaptive pruning of the deep learning model parameters.

[0093] Specifically, after constructing the parameter adaptive pruning model, a parameter-related parameter can be established. The minimization problem is solved iteratively using the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches convergence. It has a unified tensor network model (UTN model) structure, which can be represented as:

[0094]

[0095] in It is the rank of the Unified Tensor Network (UTN) model.

[0096] This minimization problem is not differentiable due to the rank(·) constraint. We can rewrite it as follows:

[0097]

[0098] in This problem is clearly non-convex and can be optimized using the Alternating Direction Multiplier Method (ADMM). Specifically, an auxiliary variable is introduced. and about Indicator function g(·), for example:

[0099]

[0100] The above formula enforces constraints The solution must be within the domain. Otherwise, the penalty is +∞, so the problem above can be rewritten as:

[0101]

[0102] Its augmented Lagrangian function can be expressed as:

[0103]

[0104] in ρ is the augmented Lagrange multiplier, and ρ>0 is the penalty parameter. Therefore, the problem can be minimized by the iterative alternating direction multiplier method (ADMM) for each subproblem.

[0105] renew The subproblem can be represented as:

[0106]

[0107] This subproblem can be solved directly by stochastic gradient descent (SGD), relative to... The partial derivatives are calculated as follows:

[0108]

[0109] therefore, Update successful:

[0110]

[0111] Where η is the learning rate. After multiple iterations, It can converge to a local minimum.

[0112] renew The subproblem can be represented as:

[0113]

[0114] By maximizing The posterior probability is obtained by minimizing the posterior probability. The posterior probability. Therefore, the above minimization problem can be rewritten as...

[0115]

[0116] The complete expression is:

[0117]

[0118] in for Its closed-form solution can be obtained by taking its partial derivative and setting the derivative to zero, as follows:

[0119]

[0120] In obtaining After adding c, we collapse it back to its original dimensions. Where Z... n Auxiliary variable The modulus-n expansion matrix, It is the modulo-n expansion matrix obtained by shrinking all factor tensors except the nth cyclic factor tensor. It is the modulo-1 expansion of all ring factor matrices except the core factor tensor after shrinking.

[0121] Other parameters can be solved within the framework of sparse Bayesian methods, where... <·> represents the expectation. Their corresponding parameters can be updated as follows: For each When i = n, we define R n,n-1 =R n-1,n ,Λ n-1,n =Λ n,n-1 n∈[1,N], and we have

[0122]

[0123]

[0124] in express r n,j -th vectors, of which

[0125]

[0126] for The update is as follows:

[0127]

[0128]

[0129]

[0130]

[0131] in Setting the value to 1e-6 results in a non-informative prior. Within the framework of sparse Bayesian learning, we can prune the zero components of these factor tensors to compress the parameters. Finally, The update can be given by the following formula:

[0132]

[0133] Finally, we can update the augmented Lagrange multipliers. By the following formula:

[0134]

[0135] Furthermore, based on the parameter adaptive pruning model that has reached convergence, irrelevant components of the parameters are pruned to determine the corresponding compressed parameters, thereby updating the parameters in the deep learning model. After the parameter update is completed, a Unified Tensor Network (UTN) model can be used to fine-tune the parameters in the deep learning model. Since the Alternating Directional Multiplier Method (ADMM) has already imposed the low-rank structure of the UTN model required for the parameters in the deep learning model onto the uncompressed deep learning model during the optimization process of minimizing the parameters, this directional decomposition does not introduce significant approximation errors, unlike the existing UTN model format and the directional decomposition in the training of the deep learning model. Typically, this fine-tuning stage is very fast, requiring only a few iterations. This is because the UTN model decomposed at the starting point of this stage already has an accuracy very close to that of the original uncompressed deep learning model.

[0136] As can be seen from the above embodiments, compared with the prior art, this application addresses the technical problems of existing deep learning models being limited by a large number of parameters, leading to high memory usage and computationally intensive processes. This application captures the low-rank structure of the parameters of each layer in the deep learning model based on a unified tensor network model, decomposing the parameters into multiple cyclic factor tensors and a core factor tensor. Based on a sparse Bayesian learning algorithm, each element in the cyclic factor tensor and the core factor tensor is assigned a Gaussian prior distribution, determining the corresponding Gaussian prior distribution results for the cyclic factor tensor and the core factor tensor. Furthermore, the noise level in the deep learning model is Gaussian-based. The prior distribution of the ma distribution is determined, and the prior distribution of the Gamma distribution corresponding to the noise level is determined. Based on the prior distributions of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor, and the prior distribution of the Gamma distribution corresponding to the noise level, a parameter adaptive pruning model is constructed. The parameter minimization problem is determined, and the minimization problem is iterated using the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches convergence. Based on the converged parameter adaptive pruning model, irrelevant components of the parameters are pruned to determine the corresponding compression parameters. This application includes, but is not limited to, the following beneficial effects:

[0137] First, this application introduces a novel tensor network model, which is a generalized version of all other tensor network models, and has more powerful representation capabilities and a low-rank structure.

[0138] Secondly, the novel tensor network model proposed in this application can adaptively learn tensor network structures within a sparse Bayesian learning framework, thus enabling it to learn corresponding tensor network representations for different real-world data.

[0139] Thirdly, this application provides an adaptive tensor network model, which compresses the parameters in the deep learning model, greatly improving the computational speed of data processing and reducing the memory required for the deep learning model. This allows the deep learning model to maintain a low number of parameters while still maintaining its high accuracy, thus avoiding affecting the accuracy of the deep learning model.

[0140] Please see Figure 3 This application provides a deep learning model parameter adaptive pruning device, comprising a parameter acquisition module 1100, a parameter decomposition module 1200, a pruning model determination module 1300, and a parameter pruning module 1400. The parameter acquisition module 1100 is configured to acquire a preset training set of a deep learning model, pre-train the deep learning model based on the training set, and determine the parameters of each layer in the deep learning model. The parameter decomposition module 1200 captures the low-rank structure of the parameters of each layer in the deep learning model based on a unified tensor network model, decomposing the parameters into multiple ring factor tensors and a core factor tensor. The pruning model determination module 1300 is configured to perform a Gaussian prior distribution on each element of the ring factor tensor and the core factor tensor based on a sparse Bayesian learning algorithm, determine the prior distribution results of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor, and prune the noise level in the deep learning model. The prior distribution of the Gamma distribution is determined by the method of flattening the noise level. Based on the prior distributions of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor, and the prior distribution of the Gamma distribution corresponding to the noise level, an adaptive parameter pruning model is constructed. The parameter pruning module 1400 is configured to determine the minimization problem of the parameters. The minimization problem is iterated using an alternating direction multiplier algorithm until the adaptive parameter pruning model reaches convergence. Based on the converged adaptive parameter pruning model, irrelevant components of the parameters are pruned to determine the corresponding compression parameters, thus completing the adaptive pruning of the deep learning model parameters.

[0141] Based on any embodiment of this application, please refer to Figure 4 Another embodiment of this application also provides an electronic device, which can be implemented by a computer device, such as... Figure 4The diagram shows the internal structure of a computer device. The computer device includes a processor, a computer-readable storage medium, a memory, and a network interface connected via a system bus. The computer-readable storage medium stores an operating system, a database, and computer-readable instructions. The database may store a sequence of control information. When executed by the processor, the computer-readable instructions enable the processor to implement a deep learning model parameter adaptive pruning method. The processor provides computational and control capabilities, supporting the operation of the entire computer device. The memory stores computer-readable instructions, which, when executed by the processor, enable the processor to execute the deep learning model parameter adaptive pruning method of this application. The network interface of the computer device is used for communication with a terminal. Those skilled in the art will understand that… Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0142] In this embodiment, the processor is used to execute... Figure 3 The system contains the specific functions of each module and its sub-modules, and the memory stores the program code and various data required to execute the aforementioned modules or sub-modules. The network interface is used for data transmission between the user terminal and the server. In this embodiment, the memory stores the program code and data required to execute all modules / sub-modules in the deep learning model parameter adaptive pruning device of this application, and the server can call the server's program code and data to execute the functions of all sub-modules.

[0143] This application also provides a storage medium storing computer-readable instructions, which, when executed by one or more processors, cause the one or more processors to perform the steps of the deep learning model parameter adaptive pruning method described in any embodiment of this application.

[0144] This application also provides a computer program product, including a computer program / instructions that, when executed by one or more processors, implement the steps of the deep learning model parameter adaptive pruning method described in any embodiment of this application.

[0145] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. This computer program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0146] The above description is only a partial embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

[0147] In summary, this application provides an adaptive tensor network model. Based on this adaptive tensor network model, the parameters in the deep learning model are compressed, which greatly improves the computational speed of data processing of the deep learning model and reduces the memory required for its operation. This allows the deep learning model to maintain a low number of parameters while still maintaining its high accuracy, thus avoiding affecting the accuracy of the deep learning model.

Claims

1. A method for adaptive pruning of deep learning model parameters, the method comprising: Includes the following steps: Obtain a training set of a preset deep learning model, pre-train the deep learning model based on the training set, and determine the parameters of each layer in the deep learning model; wherein, the training set represents image data or hyperspectral data, and the deep learning model includes an image classification model or an object detection model; The low-rank structure of the parameters of each layer in the deep learning model is captured based on the unified tensor network model, and the parameters are decomposed into multiple ring factor tensors and a core factor tensor. Based on the sparse Bayesian learning algorithm, each element in the ring factor tensor and the core factor tensor is distributed using a Gaussian prior distribution to determine the prior distribution results of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor. Similarly, the noise level in the deep learning model is distributed using a Gamma prior distribution to determine the prior distribution results of the Gamma distribution corresponding to the noise level. Based on the prior distribution results of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor, and the prior distribution results of the Gamma distribution corresponding to the noise level, a parameter adaptive pruning model is constructed. The ring factor tensor is a component in the unified tensor network model that works in conjunction with the core factor tensor, linked pairwise, and associated with specific dimension indices to capture the low-rank structure of the deep learning model parameters. order tensor; For the core factor tensor, a Gaussian prior distribution is applied to each of its corresponding elements to determine the prior distribution result of the Gaussian distribution corresponding to the core factor tensor. The Gaussian prior distribution corresponding to the core factor tensor is expressed as follows: It represents the core factor tensor. Each element conforms to a set of rules with a mean of 0 and a variance of 0. Gaussian distribution, due to the core factor tensor The prior distribution has a mean of 0. When maximizing the posterior probability, the core factor tensor... Each element also approaches 0, thus obtaining the core factor tensor. sparse representation, Represents the core factor tensor Vectorization operations, It is a scale parameter; This indicates that the first to the second are controlled respectively. The hyperparameter of the connection sparsity between the ring factor tensor and the core factor tensor; Indicates that each is determined by hyperparameters The diagonal covariance matrix formed Indicates the Kronecker product; The problem of minimizing the parameters is determined, and the minimization problem is iterated based on the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches a convergent state. Based on the convergent parameter adaptive pruning model, irrelevant components of the parameters are pruned to determine the corresponding compression parameters, thus completing the adaptive pruning of the deep learning model parameters.

2. A deep learning model parameter adaptive pruning device, characterized in that, include: The parameter acquisition module is configured to acquire a training set of a preset deep learning model, pre-train the deep learning model based on the training set, and determine the parameters of each layer in the deep learning model. The parameter decomposition module is configured to capture the low-rank structure of the parameters of each layer in the deep learning model based on the unified tensor network model, and decompose the parameters into multiple ring factor tensors and a core factor tensor; wherein, the training set represents image data or hyperspectral data, and the deep learning model includes an image classification model or an object detection model. The pruning model determination module is configured to use a sparse Bayesian learning algorithm to perform a Gaussian prior distribution on each element of the ring factor tensor and the core factor tensor, determining the prior distribution results of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor. It also performs a Gamma prior distribution on the noise level in the deep learning model, determining the prior distribution result of the Gamma distribution corresponding to the noise level. Based on the prior distribution results of the Gaussian distributions corresponding to the ring factor tensor and the core factor tensor, and the prior distribution results of the Gamma distribution corresponding to the noise level, a parameter adaptive pruning model is constructed. The ring factor tensor is a component in the unified tensor network model that works in conjunction with the core factor tensor, associated through pairwise connections and specific dimensional indices, and is used to capture the low-rank structure of the deep learning model parameters. order tensor; For the core factor tensor, a Gaussian prior distribution is applied to each of its corresponding elements to determine the prior distribution result of the Gaussian distribution corresponding to the core factor tensor. The Gaussian prior distribution corresponding to the core factor tensor is expressed as follows: It represents the core factor tensor. Each element conforms to a set of rules with a mean of 0 and a variance of 0. Gaussian distribution, due to the core factor tensor The prior distribution has a mean of 0. When maximizing the posterior probability, the core factor tensor... Each element also approaches 0, thus obtaining the core factor tensor. sparse representation, Represents the core factor tensor Vectorization operations, It is a scale parameter; This indicates that the first to the second are controlled respectively. The hyperparameter of the connection sparsity between the ring factor tensor and the core factor tensor; Indicates that each is determined by hyperparameters The diagonal covariance matrix formed Indicates the Kronecker product; The parameter pruning module is configured to determine the minimization problem of the parameters, iterate the minimization problem based on the alternating direction multiplier algorithm until the parameter adaptive pruning model reaches a convergent state, and prune irrelevant components of the parameters based on the convergent parameter adaptive pruning model to determine the corresponding compression parameters, thereby completing the adaptive pruning of the deep learning model parameters.

3. An electronic device comprising a central processing unit and a memory, characterized in that, The central processing unit is used to invoke and run a computer program stored in the memory to perform the steps of the method as described in claim 1.

4. A computer-readable storage medium, characterized in that, It stores a computer program implemented according to the method of claim 1 in the form of computer-readable instructions, which, when called by a computer, executes the steps included in the corresponding method.

Images