Model compression method and device based on structural control variables

Abstract

A model compression method and apparatus based on structural control variables are provided. The model compression method includes: introducing structural control variables into the target layer of the model to be compressed and establishing a correlation mapping relationship between the structural control variables and the corresponding layer parameter matrix; constructing an objective function based on the correlation mapping relationship, the objective function including at least a first evaluation term for constraining the performance of the model to be compressed, and a second evaluation term for applying sparsity constraints to the structural control variables using the correlation mapping relationship; training the model to be compressed using the objective function, and during the training process, updating the structural control variables through gradients to make them tend to a sparse distribution, so that the model to be compressed evolves into an intermediate model with a sparse logical structure; determining redundant channels or connections based on the numerical distribution of the structural control variables in the intermediate model; and physically deleting the model parameters corresponding to the redundant channels or connections to obtain the compressed model.

Description

Technical Field

[0001] This invention relates to artificial intelligence and computer system optimization, and more specifically, to a model compression method and apparatus based on structural control variables. Background Technology

[0002] With the rapid development of short video platforms, security analysis and understanding tasks involving massive amounts of content place higher demands on the processing capabilities of models, especially in terms of high throughput and low inference latency.

[0003] Currently, deep neural networks have been widely used in content recognition and analysis. However, they typically have a large parameter scale and high computational complexity. When actually deployed on hardware platforms such as central processing units (CPUs), graphics processing units (GPUs), and neural network processing units (NPUs), they are easily limited by computing resources and response time, making it difficult to meet real-time processing requirements.

[0004] Model compression technology is widely used to reduce the size of model parameters and computational overhead, thereby improving inference efficiency. However, in practical applications, traditional model compression methods often focus on optimization based on the number of parameters or computational cost, lacking effective collaborative modeling of model structure and parameter optimization processes. This results in structural adjustments and parameter updates being coupled during compression, making it difficult to achieve fine-grained control.

[0005] Meanwhile, different network layers play different roles in the model, and a fixed compression strategy cannot take into account the importance of each layer structure, affecting the balance between compression effect and model accuracy.

[0006] Furthermore, existing methods often fail to fully consider the actual execution characteristics of the target hardware platform. The compression process relies heavily on theoretical calculation metrics for optimization and lacks a direct feedback mechanism for key performance metrics such as inference latency. As a result, the model compression results are difficult to achieve the expected performance improvement in actual deployment.

[0007] At the same time, the lack of effective dynamic adjustment and closed-loop optimization methods during model compression makes it difficult to continuously optimize the model structure based on the running status. Summary of the Invention

[0008] The purpose of the exemplary embodiments disclosed herein is to provide a method for finely identifying and controlling redundant channels of a model to ensure a real performance improvement of the compressed model at the hardware level.

[0009] According to a first aspect of this disclosure, a model compression method based on structural control variables includes: introducing structural control variables into the target layer of the model to be compressed, and establishing a correlation mapping relationship between the structural control variables and the corresponding layer parameter matrix, so as to characterize the weight contribution of different substructures in the parameter matrix through the structural control variables; constructing an objective function based on the correlation mapping relationship, the objective function including at least a first evaluation term for constraining the performance of the model to be compressed, and a second evaluation term for applying sparsity constraints to the structural control variables using the correlation mapping relationship; training the model to be compressed using the objective function, and making the structural control variables tend to sparse distribution through gradient updates during the training process, so that the model to be compressed evolves into an intermediate model with a sparse logical structure; determining redundant channels or connections according to the numerical distribution of the structural control variables in the intermediate model; and physically deleting the model parameters corresponding to the redundant channels or connections to obtain the compressed model.

[0010] Optionally, the step of establishing the association mapping relationship between structural control variables and corresponding layer parameter matrices may include: dividing the parameter matrix into several parameter sub-blocks according to a preset physical structure dimension, wherein each parameter sub-block corresponds to an input channel or output channel in the model to be compressed; assigning a corresponding scalar to each parameter sub-block as a component of the structural control variable; and establishing the mapping relationship between the scalar and the corresponding parameter sub-block.

[0011] Optionally, the association mapping relationship can be specifically a multiplicative mapping relationship; during training, by performing a Hadamard product operation on each scalar and the corresponding parameter sub-block, the contribution of the corresponding channel to the output features of the model to be compressed is linearly adjusted using the numerical value of the scalar.

[0012] Optionally, the first evaluation term can be a task loss function that reflects the difference between the predicted value and the true label of the model to be compressed, the second evaluation term can be a sparse regularization term constructed based on the vector norm of the structural control variables, and the objective function can be a weighted fusion of the first and second evaluation terms.

[0013] Optionally, the objective function may also include a penalty factor, which serves as the weighting factor for the second evaluation term. The structural control variables are divided into a first subset of variables and a second subset of variables. The first subset of variables consists of the expected sparse parameters, and the second subset consists of fixed parameters that do not participate in sparsity. Specifically, the second evaluation term is a sparse regularization term constructed based on the expected sparse parameters, and the weight of the sparse regularization term is the penalty factor.

[0014] Optionally, the model compression method may further include: obtaining the current inference latency of the model to be compressed on the target hardware platform within a preset training interval; the construction of the objective function may further include: introducing a delay constraint term based on the current inference latency, and including the delay constraint term in the objective function, so as to optimize the structural control variables in the direction of satisfying the preset delay constraints.

[0015] Optionally, obtaining the current inference latency may include: predicting the current inference latency through a latency prediction model, or obtaining the current inference latency by deploying the current model on the target hardware platform for actual inference.

[0016] Optionally, obtaining the current inference latency may also include: within a preset time interval, comparing the inference latency predicted by the latency prediction model with the inference latency actually obtained by the target hardware platform; in response to the deviation between the two exceeding a preset fault tolerance threshold, collecting the actual inference latency as feedback and introducing it into the training and parameter correction of the model.

[0017] Optionally, the step of physically deleting model parameters corresponding to redundant channels or connections to obtain a compressed model may include: deleting corresponding model channels or connections according to the identification results to prune redundant computation paths, and obtaining a compressed model through structural reconstruction. The model compression method may further include: after obtaining the compressed model, performing computation graph optimization on the compressed model to eliminate fragmented operators caused by physically pruning redundant computation paths and improve memory access efficiency. The computation graph optimization includes: fusing continuous operators in the compressed model; merging convolution operations and normalization operations in the compressed model; and, based on the reconstructed topology of the compressed model, pre-optimizing the storage layout and access paths of the intermediate feature maps defined by the compressed model. In response to the parameter cost after the control quantity and the compressed layer are fused being lower than the parameter cost in the non-fused state, compressed layer fusion optimization is triggered, and joint storage and access optimization is performed on the corresponding intermediate feature maps.

[0018] According to a second aspect of this disclosure, a model compression device based on structural control variables includes: a processor; and a memory storing a computer program, which, when executed by the processor, performs the aforementioned model compression method.

[0019] The model compression method according to the embodiments of this disclosure can avoid the blind adjustment of structure in traditional compression methods.

[0020] The model compression method according to the embodiments of this disclosure can take into account the importance of each layer structure and balance the compression effect with the model accuracy.

[0021] The model compression method according to the embodiments of this disclosure dynamically optimizes the compression process by incorporating hardware feedback information, which can improve the running efficiency and application effect of the model in the actual deployment environment. Attached Figure Description

[0022] The above and other objects, features and advantages of exemplary embodiments of the present invention will become clearer from the following detailed description taken in conjunction with the accompanying drawings, which illustrate exemplary embodiments, wherein: Figure 1A flowchart of a model compression method according to at least one exemplary embodiment of the present disclosure is shown; Figure 2 A flowchart of a model compression method according to at least one exemplary embodiment of the present disclosure is shown; Figure 3 A flowchart illustrating a specific process for deleting model parameters according to at least one exemplary embodiment of the present disclosure is shown; Figure 4 A computational graph optimization process according to at least one example embodiment of the present disclosure is illustrated; Figure 5 A block diagram of a model compression apparatus according to at least one example embodiment of the present disclosure is shown. Detailed Implementation

[0023] The embodiments of this disclosure will now be described in detail with reference to the accompanying drawings, examples of which are illustrated in the drawings, wherein the same reference numerals always refer to the same parts. The embodiments will now be described with reference to the accompanying drawings in order to explain this disclosure.

[0024] The following detailed description is provided to aid in obtaining a full understanding of the methods, apparatus, and / or systems described herein. However, the order of operations described herein is merely illustrative and is not limited to those orders set forth herein; equivalent substitutions or changes may be made, except for operations that must occur or be performed in a specific order. Furthermore, for clarity and conciseness, descriptions of content well-known in the art will be omitted or simplified.

[0025] Unless otherwise specified, the same reference numerals generally refer to the same elements (e.g., components, steps, and methods). Reference numerals described in previous embodiments that reappear in later embodiments may be omitted. Furthermore, technical features described in different or the same embodiments can be combined in any way, as long as the combined embodiment or technical solution is complete and can solve the technical problems of this application or achieve the technical effects described or not described in this disclosure but which can be determined based on the complete technical solution described above.

[0026] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0027] A model is a parameterized mapping entity with a specific computational topology. Logically, a model is a neural network architecture (e.g., a deep neural network architecture) consisting of multiple interconnected computational layers (such as convolutional layers, fully connected layers, attention mechanism layers, etc.) used to perform nonlinear transformations from the input feature space to the output target space. From a physical and storage perspective, a model is represented as a set of structured numerical values ​​stored in computer storage media (e.g., memory, video memory, or hard disk), namely, weights (W) and biases (B) arranged in the form of matrices or tensors.

[0028] Model compression refers to the process of reducing model size and computational overhead through specific technical means while maintaining model performance (e.g., accuracy) at a relatively low level.

[0029] This disclosure establishes a learnable, structured compression mechanism by introducing structural control variables into the model parameter matrix. Instead of processing individual weights piecemeal, this disclosure divides the parameter matrix into corresponding sub-blocks (e.g., input / output channels) and establishes a multiplicative mapping relationship with the components of the structural control variables. Furthermore, this disclosure maps the model parameter matrix to computer storage resources (e.g., memory blocks or video memory units). By dividing the parameter matrix into physical sub-blocks corresponding to input or output channels and associating them with learnable structural control variables, the originally discrete topology adjustments are transformed into continuous control of the contribution to the storage dimension, thereby achieving collaborative optimization of the model's logical architecture and the underlying physical storage structure.

[0030] Furthermore, this disclosure achieves decoupling between model topology optimization and parameter updates by driving the structural control variables towards sparsity through an objective function, thus solving the problem of overly coupled and difficult-to-control structural adjustments in traditional methods. A detailed description is provided below with reference to the accompanying figures.

[0031] Figure 1 A flowchart of a model compression method according to at least one exemplary embodiment of the present disclosure is shown. Figure 2 A flowchart of a model compression method according to at least one exemplary embodiment of the present disclosure is shown. Figure 3 A flowchart illustrating a specific process for deleting model parameters according to at least one exemplary embodiment of the present disclosure is provided. Figure 4 The computational graph optimization process according to at least one example embodiment of the present disclosure is illustrated. Figure 5 A block diagram of a model compression apparatus according to at least one example embodiment of the present disclosure is shown.

[0032] Reference Figure 1 The model compression method disclosed herein may include steps S100, S200, S300, S400 and S500.

[0033] In step S100, structural control variables are introduced.

[0034] Structural control variables can be introduced into the target layer of the model to be compressed, and a mapping relationship between the structural control variables and the corresponding layer parameter matrix can be established so as to characterize the weight contribution of different substructures in the parameter matrix through structural control variables.

[0035] The model to be compressed here can be a pre-trained neural network model (e.g., a deep neural network model), and the target network layer of the model to be compressed can be determined, which may include convolutional layers, fully connected layers, or other computational layers with redundant parameters.

[0036] Convolutional layers are the most computationally intensive layers in neural networks, containing numerous channels that contribute little to feature extraction (e.g., row or column components of the model parameter matrix W). Fully connected layers have a high parameter count and require significant physical storage. Attention mechanisms, in complex models like the Transformer (e.g., Hybrid Expert (MoE) architectures), are linear layers responsible for handling weight distribution. Therefore, convolutional layers, fully connected layers, and attention mechanisms can all be target network layers for the model to be compressed.

[0037] As an example, layers with a large proportion of parameters in the model can be prioritized, such as large fully connected layers or backbone convolutional layers in deep networks. By reducing the row and column dimensions of the parameter matrix W of these layers, the static storage requirements can be directly reduced. For example, the intermediate layers of a residual network can be identified as the target network layers.

[0038] The specific process of the association mapping relationship can be as follows: First, the parameter matrix is ​​divided into several parameter sub-blocks according to the preset physical structure dimension. Each parameter sub-block corresponds to an input channel or output channel in the model to be compressed. Then, a corresponding scalar is assigned to each parameter sub-block as a component of the structural control variable. Finally, the mapping relationship between the scalar and the corresponding parameter sub-block (e.g., displacement mapping relationship) is established.

[0039] Affine transformations (e.g., dynamic scaling using scaling factors and offset terms) and gating (e.g., introducing lightweight decision networks to calculate gate control in real time based on current features and automatically shutting down redundant channels through gating) can be used to establish mapping relationships.

[0040] In one example, the structural control variable S can be represented as a scaling factor or gating parameter corresponding one-to-one with a channel or weight set. Its initial value can be set to all 1s or initialized based on the pre-trained weight distribution. The structural control variable can participate in training along with the model parameters and be updated through backpropagation, thereby enabling the model to dynamically adjust its structure during training. The aforementioned correlation mapping can be specifically a multiplicative mapping. During training, the contribution of each scalar to the output features of the model to be compressed can be linearly adjusted by performing a Hadamard product operation on each scalar and its corresponding parameter sub-block, utilizing the magnitude of the scalar value.

[0041] Specifically, the neural network model can be represented by the following equation (1), and after introducing the structural control variable S, the neural network model can be represented by the following equation (2).

[0042] (1) (2) In equations (1) and (2), The output of the neural network model. For neural network model functions; Input data for a neural network model; The weight parameters of a neural network model (also called model parameters or parameters). : Structural control variables (matrix form) control the sparsity and structure of the model weights; The Hadamard product is the element-wise multiplication of a matrix or vector. This allows the introduction of structural control variables for the target network layer, representing the importance of each channel, convolutional kernel, or weight reassembly.

[0043] With the introduction of structural control variables, W can be regarded as the base weight layer, which is used to perform the original feature transformation operation. The structural control variable S can be the gate weight layer, which is used to adjust the weights of each channel or connection. The output of the gate weight layer can be multiplied with the output of the base weight layer channel by channel or element by element, thereby controlling the effectiveness of the corresponding connection.

[0044] In one example, when the gate weights approach zero, the corresponding connections are considered redundant; when their values ​​are large, it indicates that the connection contributes significantly to the model output. This structural decoupling separates the model's expressive power from its structural control capabilities, providing a foundation for subsequent compression.

[0045] The calculation of a certain layer of a neural network model can be expressed as the following equation (3): (3) in, This is the output of this layer (the result after activation function and structure control). Activation functions (most commonly Sigmoid, ReLU, etc.) introduce nonlinearity into the computation, allowing the model to learn complex patterns. B S is the base weight matrix of this layer, and S is the structural control variable (represented in matrix form, 2D). This represents the Hadamard product (element-wise multiplication), which is the multiplication of corresponding elements of two matrices.

[0046] In step S200, an objective function is constructed, for example, an objective function can be constructed based on an association mapping relationship.

[0047] The objective function may include at least a first evaluation term for constraining the performance of the model to be compressed, and a second evaluation term for applying sparsity constraints to the structural control variables using the correlation mapping relationship.

[0048] The objective function is constructed in the above manner. The first evaluation term ensures that the model's performance does not drastically degrade, while the second evaluation term uses the mapping relationship to induce sparsity in the structural control variables, compressing the S-components of redundant channels to near zero. The first evaluation term can be a task loss function reflecting the difference between the predicted values ​​and the true labels of the model to be compressed, and the second evaluation term can be a sparse regularization term constructed based on the vector norm of the structural control variables. However, the specific implementation of the first and second evaluation terms is not limited to these; the objective function may include other evaluation terms in addition to the first and second evaluation terms.

[0049] As an example, the second evaluation term, serving as the structured sparsity regularization term, can be constructed based on the group norm to achieve channel-level or structure-level sparsity. Furthermore, a continuous scheduling function can be introduced to dynamically adjust the strength of the sparsity constraint, making the sparsity constraint weaker in the early stages of training and gradually stronger in the later stages, thereby preventing the model from prematurely losing important structures in the initial phase. Through these methods, a smooth transition from a dense structure to a sparse structure can be achieved.

[0050] As an example, the objective function can be a weighted fusion of the first and second evaluation terms. In the early stages of training, the weight of the first evaluation term can be set to the highest to ensure that the model parameters W learn the core features; subsequently, the weight of the second evaluation term (sparse constraint) is gradually increased to guide the structural control variable S to identify redundant channels. Additionally, the objective function can include a penalty factor, which can serve as the weighting factor for the second evaluation term. The structural control variable can be divided into a first subset and a second subset. The first subset consists of the expected sparse parameters, and the second subset consists of fixed parameters that do not participate in sparsity. Specifically, the second evaluation term can be a sparse regularization term constructed based on the expected sparse parameters, with the weight of the sparse regularization term being the penalty factor.

[0051] Specifically, the overall loss function is defined as follows: (4) in, It is a sparse regularization term. It is a penalty factor. It can be solved by minimizing the objective function and combining it with some optimization solvers.

[0052] As described above, this disclosure can control structural variables. It is divided into two parts: the expected sparse parameters. By including parameters that do not participate in sparsity, the tedious process of model compression, such as the need for further fine-tuning after model pruning, can be avoided. Therefore, a combination of initial presets and automatic selection during training can be used. For example, the expected sparse parameters and the parameters that do not participate in sparsity can be predefined, and then the structural control variable S can be dynamically updated during training using the objective function. For example, the energy distribution of each layer can be initially evaluated using L1 or L2 norms. For layers with smaller norm mean values ​​and exhibiting high redundancy, a higher compression quota can be pre-allocated; conversely, more dimensions can be retained.

[0053] In one example, the parameter matrix L can be used. 21 As a sparsity regularization, the norm, for the control variable matrix S, when sparse regularization is added to both rows and columns, the sparse term can be expressed as the following equation (5): (5) in, Let S be the number of rows in the control variable matrix S of the i-th layer structure that need to be diluted when expanded by rows. Let S be the number of columns in the control variable matrix S of the i-th layer structure that need to be diluted when expanded. This represents the number of model layers.

[0054] In one example, the sparsity penalty coefficient in the objective function can increase smoothly in an S-shaped curve as training progresses, guiding the structural control variable S to focus on preserving feature information in the early stages of training and on inducing sparsity in physical sub-blocks in the later stages. Alternatively, an iterative algorithm incorporating a threshold operator can be used to perform shrinkage processing based on the energy distribution of the row or column parameters of the structural control variable S, causing the components of the structural control variable S corresponding to parameter sub-blocks that meet preset conditions to be zeroed, thereby determining the physical dimensions to be pruned.

[0055] Specifically, this disclosure may use a soft threshold iterative algorithm, with row parameters... Update using equations (6) and (7) below.

[0056] (6) (7) in, Let be the learning rate. Updating the column parameters of the structural control variable S only requires matrix transposition, and is also done using the soft thresholding iterative algorithm.

[0057] Using the soft threshold iterative algorithm described above, if the norm of a certain row is greater than the set threshold... This indicates that the channel contains important information, so its value can be slightly reduced but retained. If the norm is less than the threshold, it can be forced to 0.

[0058] In one example, an S-shaped, smoothly increasing penalty factor, i.e., a penalty coefficient, can be introduced. from Gradually increase to the preset maximum value Specifically, it can be expressed as the following formula (8). (8) In equation (8), k represents the steepness of the control curve. The larger k is, the faster the coefficient increases and the more concentrated the rising phase becomes. At the midpoint of the curve, the coefficient increases most rapidly in this step. However, the variation of the sparsity penalty intensity with time t is not limited to that shown in equation (8).

[0059] In step S300, the objective function is used to train the model to be compressed. During the training process, the structural control variables are updated by gradient to make them tend to be sparsely distributed, so that the model to be compressed evolves into an intermediate model with a sparse logical structure.

[0060] During the iterative training of the model based on the constructed objective function, the structural control variable S can gradually change with gradient updates, causing the weights of some connections to gradually decrease until they approach zero.

[0061] As an example, different sparsity constraint strengths can be assigned to different network layers based on parameter sensitivity analysis results (e.g., Lp norm or other importance metrics) to achieve finer structure optimization. Through continuous training, the model gradually forms a compressed form with a sparse structure while maintaining high task performance.

[0062] For details, please refer to Figure 2 During model training, a hardware delay feedback mechanism can be introduced to guide the optimization direction of the model structure.

[0063] The model compression method according to embodiments of this disclosure may further include step S310, in which hardware inference latency feedback may be introduced. For example, within a preset training interval, the current inference latency of the model to be compressed on the target hardware platform may be obtained.

[0064] The preset training interval here can be the sampling period for synchronizing performance information between the model and the target hardware platform (which may include at least one of CPU, GPU, and NPU). The preset training interval can be defined in units of training iterations or epochs.

[0065] In the early stages of training, since the model focuses on feature learning and has a low penalty coefficient, a larger sampling interval can be set to reduce the computational overhead caused by deployment testing. In the middle stages of training, the interval can be shortened. Hardware-measured latency or hardware-predicted latency can accurately guide the optimization direction of the objective function, ensuring that structural control variables are pruned iteratively within a preset region, ultimately achieving model compression.

[0066] By introducing hardware inference delay feedback, a delay constraint term based on the current inference delay can be included when constructing the objective function, and this delay constraint term can be factored into the objective function to optimize the structural control variables in a direction that satisfies the preset delay constraints. Therefore, the inference delay can be used as a control signal in the update process of the structural control variables.

[0067] Specifically, a delay function can be introduced: (9) Constructing Constraint Loss: (10) In equations (9) and (10), W and S represent the model parameters and structural control variables, respectively. is the total loss function (the optimization objective during model training); L is the basic loss term, which can be determined by the above equation (4). The weighting coefficients for latency / overhead constraints control the strength of the additional penalty terms, exceeding... The time is relatively larger, lower than Relatively smaller (penalty can be 0); T is the current inference latency of the model; T target The target overhead constraint can be represented as an upper limit on the runtime on the target hardware platform. The constraint loss method based on hardware inference latency feedback described above is merely an example, and this disclosure is not limited thereto.

[0068] The current inference latency of the current model on the target hardware platform can be predicted by the latency prediction model, or the current inference latency can be obtained by deploying the current model on the target hardware platform for actual inference.

[0069] Latency prediction models can be built offline based on real-world hardware test data. For example, a large number of operators of different sizes and types can be run on the target hardware platform, their actual execution times recorded, and a feature database constructed by combining this data with performance parameters such as the number of computing cores and memory access bandwidth. Subsequently, a lightweight regression model is trained using this test data, taking the model's parameters and hardware model as input and outputting the predicted latency. By continuously optimizing this predictor to simulate the operating characteristics of real hardware, real-time latency feedback can be provided to the total loss function during training, guiding model evolution. This can significantly improve training efficiency and reduce time costs.

[0070] During model reconstruction, if the model needs to be deployed on the target hardware platform for inference every time the structural control variable S is updated, the training process will be frequently interrupted. However, the prediction model can return the results quickly, keeping the training process highly consistent.

[0071] Furthermore, introducing hardware inference latency feedback enables hardware-aware automated model reconstruction. Compared to actual deployment, this allows for precise pruning of high-latency physical sub-blocks while maintaining model accuracy, thus better adapting to the target hardware platform and improving inference speed.

[0072] Obtaining inference latency by combining prediction with actual deployment inference can balance the efficiency and accuracy of the reconstruction process. For example, actual deployment inference can be performed when the differences in inference latency prediction results from multiple latency prediction models are too large, thus obtaining the actual inference latency; alternatively, the difference between historical statistical inference latency and predicted inference latency can be used to determine whether the current model needs to be deployed for inference, obtaining the actual inference latency. However, this is merely an example, and this disclosure is not limited to this. For instance, if the difference between the predicted inference latency and historical inference latency is too large, the current model can be deployed on the target hardware platform for actual inference, obtaining the actual inference latency as the current inference latency. Therefore, in response to the difference between the inference latency predicted by the latency prediction model and historical inference latency exceeding a preset threshold, the current model can be deployed on the target hardware platform for actual inference, obtaining the actual inference latency as the current inference latency. Alternatively, a combination of the above methods can be used to determine whether the current model needs to be deployed for inference, obtaining the actual inference latency, and using this as feedback to guide model compression. Furthermore, to reduce the system overhead caused by real-time monitoring, a periodic sampling mechanism can be adopted to compare and analyze the inference latency output by the prediction model with the actual inference latency of the hardware platform at fixed time intervals. When the deviation between the two exceeds a preset fault tolerance threshold, actual inference latency data (e.g., multiple sets of actual inference latency data) can be automatically collected and used as feedback samples to introduce into the model training and parameter correction process, so as to achieve continuous optimization of the prediction model and improvement of prediction accuracy.

[0073] Reference Figure 1 and Figure 2 In step S400, redundant channels (and / or connections) are identified. Specifically, redundant channels or connections can be determined based on the numerical distribution of structural control variables in the intermediate model. Here, the intermediate model refers to the model that has not yet been reconstructed after training.

[0074] The values ​​of structural control variables S, which correspond one-to-one with each physical channel and connection path, can be extracted from the intermediate model after training. By setting a preset threshold as the judgment boundary, the physical channels or connections corresponding to variables with values ​​lower than the preset threshold are identified as redundant parts, providing a basis for subsequent structural reconstruction.

[0075] In step S500, the model parameters corresponding to redundant channels or connections are physically deleted to obtain a compressed model.

[0076] Reference Figure 3 The step of physically deleting model parameters corresponding to redundant channels or connections to obtain a compressed model may include steps S510 and S520.

[0077] After identifying components in the structural control variable whose values ​​are below a preset threshold, or after determining that the value of the structural control variable S is below a preset threshold, in step S510, the corresponding model channels or connections can be deleted based on the identification results. For example, the corresponding rows or columns can be removed from the weight tensor to achieve channel-level physical pruning.

[0078] As an example, based on the differences in the sensitivity of different layers to accuracy, the independent decision boundaries of each layer can be automatically calculated using local numerical distributions. For instance, the criteria can be automatically relaxed in networks with more redundancy, while stricter retention principles can be maintained in key feature extraction layers. In addition to pruning through preset thresholds, other methods can also be used for reconstruction.

[0079] In step S520, a compressed model is obtained through structural reconstruction. This structural reconstruction may include path pruning and structural merging.

[0080] For redundant computational paths, the computation graph topology can be modified to skip invalid operator branches and re-stitch the remaining active nodes. Alternatively, after removing redundancy, active nodes can be re-stitched, transforming the logically sparse model into a physically compressed model structure. Therefore, the above process involves not only logically zeroing out values ​​but also permanently releasing memory space, thereby eliminating the memory access and computational overhead of redundant parameters, ultimately obtaining a compact compressed model that perfectly matches the computing power of the target hardware platform.

[0081] In addition, after obtaining the compressed model, computation graph optimization can be performed on the compressed model to eliminate fragmented operators caused by physical pruning of redundant computation paths and improve memory access efficiency.

[0082] For details, please refer to Figure 4 The computational graph optimization may include steps S610, S620 and S630.

[0083] In step S610, continuous operators in the compressed model are fused. Normalization parameters can be pre-integrated into the weights and biases of the convolution kernel through algebraic transformations, ensuring that only one linear operation is performed during inference. Furthermore, continuous linear or nonlinear operators in the computation graph can be identified, and the activation function can be placed directly at the computation output, eliminating frequent cross-references between memory and cache for intermediate feature maps. This reduces the memory access frequency during model inference, ultimately achieving higher bandwidth utilization and efficient operation on the target hardware platform.

[0084] In step S620, the convolution and normalization operations in the compressed model are merged. For example, parameters such as the mean, variance, scaling factor, and offset of the normalization layer can be pre-fused into the weights and biases of the convolutional layer. During the inference phase, the convolution and normalization operations that originally needed to be performed sequentially are merged into a single linear operation, eliminating the previously independent normalization layer in the computation graph. This reduces parameter redundancy in the model and eliminates the I / O overhead of intermediate feature maps in memory, thereby reducing inference latency and enabling the compressed model to achieve higher bandwidth utilization and efficient operation on the target hardware platform.

[0085] In step S630, the storage layout and access paths of the intermediate feature maps defined by the compressed model can be pre-optimized based on the reconstructed topology of the compressed model. For example, the lifetime and memory usage of each layer of feature maps during inference can be accurately calculated based on the reconstructed network topology. Since redundant channels have been physically eliminated, the size of the intermediate feature maps is significantly reduced. In addition, the data arrangement format in memory is optimized to ensure the continuity of memory access, taking into account the caching characteristics of the hardware. This maximizes bandwidth utilization and reduces memory access latency by reducing discontinuous memory access operations and redundant I / O reads and writes. Furthermore, the storage layout and access paths of the intermediate feature maps can be pre-optimized based on the topology formed by the reconstructed compressed model. If the parameter cost after the control quantity and the compressed layer are fused is lower than the parameter cost in the non-fused state, the compressed layer fusion optimization is triggered, and joint storage and access optimization is implemented for the corresponding intermediate feature maps to reduce memory access bandwidth pressure and improve hardware inference efficiency. The specific process of the above-described computation graph optimization is merely an example, but this disclosure is not limited thereto. In addition, the compressed model can be deployed directly without computation graph optimization.

[0086] Reference Figure 5The model compression apparatus based on structural control variables according to at least one embodiment of the present disclosure may include a processor 1100 and a memory 1200. The memory 1200 may store a computer program that, when executed by the processor, performs the model compression method described above.

[0087] The above has been referred to Figures 1 to 5 A model compression method and apparatus according to exemplary embodiments of the present disclosure are described. However, it should be understood that the devices, units, apparatuses, etc., shown in the drawings can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these units and devices may correspond to dedicated integrated circuits, pure software code, or modules combining software and hardware. Furthermore, one or more functions implemented by these systems or apparatuses may also be uniformly executed by components in a physical entity device (e.g., a processor, client, or server).

[0088] The instructions stored in the aforementioned computer-readable storage medium can be executed in environments deployed in computer devices such as clients, hosts, agent devices, and servers. It should be noted that the instructions can also be used to perform additional steps beyond those described above, or to perform more specific processing while executing the aforementioned steps. The details of these additional steps and further processing are already provided in the reference... Figures 1 to 5 The relevant equipment and methods have been mentioned in the description of the process, so they will not be repeated here to avoid repetition.

[0089] It should be noted that the process generation method according to the exemplary embodiments of this disclosure can rely entirely on the operation of computer programs or instructions to achieve the corresponding functions. That is, each device corresponds to each step in the functional architecture of the computer program, so that the entire system is called through a special software package (e.g., a lib library) to achieve the corresponding functions.

[0090] In addition, this disclosure also provides a non-transitory computer-readable storage medium storing a computer program that performs the above-described model compression method when the computer program is executed.

[0091] The model compression method according to embodiments of this disclosure can finely identify and control redundant channels of the model to ensure the real performance improvement of the compressed model at the hardware level.

[0092] The model compression method according to the embodiments of this disclosure can avoid the blind adjustment of structure in traditional compression methods.

[0093] The model compression method according to the embodiments of this disclosure can take into account the importance of each layer structure and balance the compression effect with the model accuracy.

[0094] The model compression method according to the embodiments of this disclosure dynamically optimizes the compression process by incorporating hardware feedback information, which can improve the running efficiency and application effect of the model in the actual deployment environment.

[0095] While specific terminology has been used to describe various embodiments of this disclosure, the specification and drawings are to be regarded as illustrative rather than restrictive in order to aid in understanding this disclosure. Various modifications and changes can be made by those skilled in the art (e.g., different features in different embodiments may be combined) without departing from the broader spirit and scope of this disclosure. Therefore, the scope of this disclosure is not limited by the specific embodiments and examples, but by the claims and their equivalents.

Claims

1. A model compression method based on structural control variables, characterized in that, include: Introduce structural control variables into the target layer of the model to be compressed, and establish a mapping relationship between the structural control variables and the corresponding layer parameter matrix, so as to characterize the weight contribution of different substructures in the parameter matrix through the structural control variables; An objective function is constructed based on the correlation mapping relationship. The objective function includes at least a first evaluation term for constraining the performance of the model to be compressed, and a second evaluation term for applying sparsity constraints to the structural control variables using the correlation mapping relationship. The model to be compressed is trained using the objective function. During the training process, the structural control variables are updated through gradients to make them tend to be sparsely distributed, so that the model to be compressed evolves into an intermediate model with a sparse logical structure. Redundant channels or connections are determined based on the numerical distribution of structural control variables in the intermediate model; Physically delete the model parameters corresponding to the redundant channels or connections to obtain the compressed model. The model compression method further includes: Within a preset training interval, the current inference latency of the model to be compressed on the target hardware platform is obtained; the construction of the objective function further includes: introducing a latency constraint term based on the current inference latency, and incorporating the latency constraint term into the objective function, so that the structural control variables are optimized in a direction that satisfies the preset latency constraints. Obtaining the current inference delay includes: The current inference latency is predicted by a latency prediction model, or the current inference latency is obtained by deploying the current model on the target hardware platform for actual inference. Within a preset time interval, the inference latency predicted by the latency prediction model is compared with the actual inference latency obtained by the target hardware platform. If the deviation between the two exceeds a preset fault tolerance threshold, the actual inference latency is collected as feedback and used for model training and parameter correction. The physical deletion of model parameters corresponding to the redundant channels or connections to obtain the compressed model includes: Based on the identification results, corresponding model channels or connections are deleted to prune redundant computational paths, and a compressed model is obtained through structural reconstruction. The model compression method further includes: after obtaining the compressed model, performing computation graph optimization on the compressed model to eliminate fragmented operators caused by physical pruning of redundant computation paths and improve memory access efficiency, wherein the computation graph optimization includes: The continuous operators in the compression model are fused together; The convolution and normalization operations in the compressed model are combined; and, Based on the reconstructed topology of the compression model, the storage layout and access paths of the intermediate feature maps defined by the compression model are pre-optimized. Specifically, in response to the fact that the parameter cost after the control quantity and the compression layer are fused is lower than the parameter cost in the non-fused state, the compression layer fusion optimization is triggered, and joint storage and access optimization is performed on the corresponding intermediate feature map.

2. The model compression method based on structural control variables according to claim 1, characterized in that, Establishing the association mapping relationship between the structural control variables and the corresponding layer parameter matrices includes: The parameter matrix is ​​divided into several parameter sub-blocks according to a preset physical structure dimension, wherein each parameter sub-block corresponds to an input channel or an output channel in the model to be compressed. Assign a corresponding scalar to each of the parameter sub-blocks as a component of the structure control variable; Establish a mapping relationship between the scalar and the corresponding parameter sub-block.

3. The model compression method based on structural control variables according to claim 2, characterized in that, The association mapping relationship is specifically a multiplicative mapping relationship; During training, the contribution of the corresponding channel to the output features of the model to be compressed is linearly adjusted by performing a Hadamard product operation on each scalar and the corresponding parameter sub-block, using the numerical value of the scalar.

4. The model compression method based on structural control variables according to claim 1, characterized in that, The first evaluation term is a task loss function that reflects the difference between the predicted value and the true label of the model to be compressed, the second evaluation term is a sparse regularization term constructed based on the vector norm of the structural control variable, and the objective function is a weighted fusion of the first evaluation term and the second evaluation term.

5. The model compression method based on structural control variables according to claim 4, characterized in that, The objective function also includes a penalty factor, which serves as a weighting factor for the second evaluation term. The structural control variables are divided into a first subset of variables and a second subset of variables. The first subset of variables consists of expected sparse parameters, and the second subset of variables consists of fixed parameters that do not participate in sparsity. The second evaluation term is specifically a sparse regularization term constructed based on the expected sparse parameters, and the weight of the sparse regularization term is the penalty factor.

6. A model compression device based on structural control variables, comprising: processor; A memory storing a computer program that, when executed by a processor, performs the model compression method according to any one of claims 1 to 5.

Images