Image classification model compression, application method and medium combining convolution channel and convolution layer shrinkage

Abstract

The application discloses an image classification model compression method combining convolution channel and convolution layer shrinkage, an application method and a medium, and relates to the field of artificial intelligence. The method comprises the following steps: a channel shrinkage module is connected in series after a first convolution layer of a residual block of an image classification model, and a layer shrinkage module is connected in series after a second convolution layer; in the training process, the channel shrinkage module and the layer shrinkage module are used to record the importance of each residual block and the importance of each output channel of each convolution layer in each residual block, so that the image classification model is reconstructed according to the importance of the residual block and the importance of each channel of the convolution layer. The application can dynamically cut the redundant layers and inefficient channels in the image classification model, retain the network structure with a high influence degree on the image classification task, effectively reduce the structural complexity of the model, reduce the number of parameters of the model, and greatly reduce the calculation amount of the model under the condition that the performance of the compressed model remains basically unchanged.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to an image classification model compression, application method, and medium that combines convolutional channels and convolutional layer shrinkage. Background Technology

[0002] In recent years, deep neural networks have made groundbreaking progress in fields such as computer vision and natural language processing. However, their massive number of parameters and computational costs severely limit their deployment on embedded and miniaturized devices. To address this challenge, image classification model compression techniques based on deep neural networks have gradually become a research hotspot. The core objective is to reduce the complexity of image classification models through network parameter simplification and structural optimization, while maintaining the performance of the original model as much as possible. Traditional pruning methods rely on manually set thresholds or heuristic rules, making it difficult to achieve refined evaluation of the differences in importance between layers. Furthermore, research on the synergistic mechanism between whole-layer structure compression and intra-layer channel-level compression is insufficient, and global compression easily leads to the degradation of key feature representation capabilities. Therefore, existing image classification model compression techniques have certain shortcomings. Summary of the Invention

[0003] The purpose of this application is to provide a method and medium for compressing and applying an image classification model that combines convolutional channels and convolutional layer shrinkage. This method can effectively reduce the structural complexity of the model, reduce the number of parameters, and significantly reduce the computational load while ensuring that the performance of the compressed image classification model remains basically unchanged, thus achieving lightweight model processing.

[0004] To achieve the above objectives, this application provides the following solution:

[0005] In a first aspect, this application provides an image classification model compression method that combines convolutional channels and convolutional layer shrinkage, including:

[0006] Obtain an initial image classification model; the initial image classification model includes a fixed start network structure, a fixed end network structure, and several residual blocks located between the start network structure and the end network structure; the residual blocks include two convolutional layers;

[0007] A channel shrinking module is inserted after the first convolutional layer of each residual block, and a layer shrinking module is inserted after the second convolutional layer to obtain an improved image classification model. The channel shrinking module is used to calculate a first adaptive threshold using a soft thresholding method, dynamically select data with smaller absolute values ​​between the first adaptive threshold and the input feature values ​​of the channel shrinking module for output, and obtain the importance of each output channel in the first convolutional layer. The layer shrinking module is used to calculate a second adaptive threshold using a soft thresholding method, dynamically select data with smaller absolute values ​​between the second adaptive threshold and the input feature values ​​of the layer shrinking module for output, and obtain the importance of each output channel in the second convolutional layer and the importance of its corresponding residual block.

[0008] The improved image classification model is trained using the target image training set. During the training process, the importance of each residual block and the importance of each output channel of each convolutional layer in each residual block are recorded at multiple preset iterations.

[0009] The initial image classification model is reconstructed based on the importance of each residual block corresponding to multiple preset iterations and the importance of each output channel of each convolutional layer in each residual block, resulting in a reconstructed image classification model; the reconstructed image classification model is used to classify the image to be processed.

[0010] Secondly, this application provides an application method for an image classification model, including:

[0011] Obtain the image to be processed;

[0012] The image to be processed is input into the image classification model to obtain the image classification result; the image classification model is a reconstructed image classification model based on the above-mentioned image classification model compression method that combines convolutional channels and convolutional layer shrinkage.

[0013] Thirdly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image classification model compression method combining convolutional channels and convolutional layer shrinkage, or the above-described image classification model application method.

[0014] According to the specific embodiments provided in this application, the following technical effects are disclosed:

[0015] This application provides a method and medium for compressing and applying an image classification model that combines convolutional channel and convolutional layer shrinkage. A channel shrinkage module is inserted after the first convolutional layer of the residual block in the image classification model, and a layer shrinkage module is inserted after the second convolutional layer. The channel shrinkage module calculates a first adaptive threshold using a soft thresholding method, dynamically selecting data whose absolute value is smaller between the first adaptive threshold and the input feature value of the channel shrinkage module for output, and obtaining the importance of each output channel in the first convolutional layer. The layer shrinkage module calculates a second adaptive threshold using a soft thresholding method, dynamically selecting data whose absolute value is smaller between the second adaptive threshold and the input feature value of the layer shrinkage module for output, and obtaining the importance of each output channel in the second convolutional layer and the importance of its corresponding residual block. During the training of the image classification model with the channel and layer shrinkage modules inserted, the channel and layer shrinkage modules record the importance of each residual block and the importance of each output channel of each convolutional layer in each residual block at multiple preset iterations, thereby reconstructing the image classification model based on the importance of the residual blocks and the importance of each channel in the convolutional layers. This application introduces a learnable structural importance evaluation module (i.e., a channel shrinking module and a layer shrinking module) to dynamically prune redundant layers and inefficient channels in the image classification model, retaining the network structure that has a high impact on the image classification task. This effectively reduces the structural complexity of the model, the number of model parameters, and the computational load of the model while ensuring that the performance of the compressed model remains basically unchanged, thus achieving lightweight model processing. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of an image classification model compression method combining convolutional channels and convolutional layer shrinkage in one embodiment of this application;

[0018] Figure 2 A flowchart illustrating an image classification model compression method combining convolutional channels and convolutional layer shrinkage, provided as an embodiment of this application;

[0019] Figure 3 This is a schematic diagram of the PruneNet18 network structure provided in an embodiment of this application;

[0020] Figure 4 This is a schematic diagram of a residual block structure provided in an embodiment of this application;

[0021] Figure 5 This is a schematic diagram of a channel contraction module structure provided in an embodiment of this application;

[0022] Figure 6 This is a schematic diagram of a layer shrinkage module structure provided in an embodiment of this application;

[0023] Figure 7 This is a schematic diagram of the structure of the first and second multi-value mapping modules provided in an embodiment of this application;

[0024] Figure 8 This is a schematic diagram of a single-value mapping module structure provided in an embodiment of this application;

[0025] Figure 9 This is a schematic diagram illustrating the training process of an image classification model with added channel shrinkage module and layer shrinkage module according to an embodiment of this application;

[0026] Figure 10 A schematic diagram illustrating the reconstruction process of an image classification model provided in an embodiment of this application;

[0027] Figure 11 A schematic diagram of the starting structure of an image classification model provided in an embodiment of this application;

[0028] Figure 12 A flowchart illustrating the process of adding corresponding structures and parameters based on an importance list according to an embodiment of this application;

[0029] Figure 13 A schematic diagram of the basic structure of a reconstructed network provided in an embodiment of this application;

[0030] Figure 14 This is a schematic diagram of the basic structure of the reconstructed network provided in one embodiment of this application;

[0031] Figure 15 This is a schematic diagram of the end structure of an image classification model provided in an embodiment of this application;

[0032] Figure 16 This is a flowchart illustrating an application method of an image classification model provided in an embodiment of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] The purpose of this invention is to provide an image classification model compression, application method, and medium that combines convolutional channel and convolutional layer shrinkage. The compression method belongs to a deep neural network compression method with soft thresholding as its core. By using the importance values ​​of convolutional channels and convolutional layers obtained during training, adaptive model compression is achieved. The network is reconstructed based on the importance values ​​of convolutional channels and convolutional layers, which significantly reduces the structural complexity and number of parameters of the compressed model.

[0035] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0036] The image classification model compression method combining convolutional channels and convolutional layer shrinkage provided in this application can be applied to, for example... Figure 1 The application environment shown is as follows. The terminal communicates with the server via a network. A data storage system stores the data the server needs to process. This data storage system can be set up independently, integrated into the server, or located in the cloud or on another server. The terminal can send the initial image classification model to be compressed to the server. After receiving the initial image classification model, the server inserts a channel shrinking module after the first convolutional layer of each residual block in the initial image classification model, and a layer shrinking module after the second convolutional layer, to obtain an improved image classification model. The improved image classification model is then trained using a target image training set. During training, the importance of each residual block and the importance of each output channel of each convolutional layer in each residual block are recorded at multiple preset iteration counts. Based on the importance of each residual block at multiple preset iteration counts and the importance of each output channel of each convolutional layer in each residual block, the initial image classification model is reconstructed to obtain a reconstructed image classification model. This reconstructed image classification model is used to classify the image to be processed. The server can then feed back the obtained reconstructed image classification model to the terminal. Furthermore, in some embodiments, the image classification model compression method combining convolutional channels and convolutional layer shrinkage can also be implemented separately by the server or the terminal. For example, the terminal can directly compress the initial image classification model to be compressed by combining convolutional channels and convolutional layer shrinkage, or the server can obtain the initial image classification model to be compressed from the data storage system and perform image classification model compression by combining convolutional channels and convolutional layer shrinkage.

[0037] The terminal can be, but is not limited to, various desktop computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. The server can be a standalone server or a server cluster consisting of multiple servers, or it can be a cloud server.

[0038] In one exemplary embodiment, such as Figure 2 As shown, an image classification model compression method combining convolutional channels and convolutional layer shrinkage is provided. This method is executed by a computer device, specifically by a terminal or server alone, or by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 The following steps, 101 to 104, are used as an example to illustrate the process of using a server in the example.

[0039] Step 101: Obtain an initial image classification model; the initial image classification model includes a fixed starting network structure, a fixed ending network structure, and several residual blocks located between the starting and ending network structures; each residual block includes two convolutional layers. The image classification model is a deep neural network structure.

[0040] This embodiment uses PruneNet18, a network built on the ResNet18 network, as an example. The deep network framework used is a batch of images containing k categories of objects. For input, a single image contains only one type of object, where B represents the number of images input into the network in the same batch, and img n This represents the nth image in this batch of images. Each image has C channels, height, and width, respectively. Before being input into the network, the number of channels, height, and width of each image are preprocessed to be 3, 32, and 32, respectively. Therefore Will As input to the network, the image is ultimately classified k-fold based on the object categories in the image to obtain the prediction result.

[0041] Figure 3 The PruneNet18 network structure is shown. The PruneNet18 network includes eight serial residual blocks located between the start and end network structures. As an example, the start network structure includes... Figure 3 The network consists of convolutional layers (starting convolutional layers), normalization layers (starting normalization layers), and activation layers (starting activation layers). The terminal network structure includes... Figure 3 The pooling layer (end pooling layer), dimensionality reduction layer, and linear layer (end linear layer) in the model.

[0042] Step 102: A channel shrinking module is inserted after the first convolutional layer of each residual block, and a layer shrinking module is inserted after the second convolutional layer to obtain an improved image classification model. The channel shrinking module is used to calculate a first adaptive threshold using a soft thresholding method, dynamically select data with smaller absolute values ​​between the first adaptive threshold and the input feature values ​​of the channel shrinking module for output, and obtain the importance of each output channel in the first convolutional layer. The layer shrinking module is used to calculate a second adaptive threshold using a soft thresholding method, dynamically select data with smaller absolute values ​​between the second adaptive threshold and the input feature values ​​of the layer shrinking module for output, and obtain the importance of each output channel in the second convolutional layer and the importance of its corresponding residual block.

[0043] Step 103: Train the improved image classification model using the target image training set, and during the training process, record the importance of each residual block and the importance of each output channel of each convolutional layer in each residual block at multiple preset iterations.

[0044] The target image training set is determined based on the actual image classification requirements. As an example, if image classification is applied to autonomous driving scenarios, the target image training set is constructed using image data from traffic scenarios.

[0045] During the training process, Figure 3 The processing procedure for the PruneNet18 network structure shown is as follows:

[0046] First, the image features are initially extracted using the starting convolutional layer (7×7 kernel, stride 2, padding 3, output channels 64), and the feature map is then fed into the starting normalization layer for processing.

[0047] Second, the feature map is standardized using the initial normalization layer to make the data have a relatively stable distribution. The normalized feature map is then fed into the initial activation layer, which uses the ReLU function to introduce non-linear characteristics into the features, accelerate the training process, and improve the model's performance. The processed feature map is then fed into subsequent modules for further processing.

[0048] Third, deeper feature extraction is performed using eight concatenated residual blocks. At the same time, the importance information of each network structure is obtained by using the channel shrinking module and layer shrinking module in the residual blocks. Finally, the extracted classification feature information is input into the subsequent modules.

[0049] Fourth, the classification feature information is mapped to classification confidence information by using the initial pooling layer, dimensionality reduction layer and initial linear layer, which is then used as the final output of the network.

[0050] Step 104: The initial image classification model is reconstructed based on the importance of each residual block corresponding to multiple preset iterations and the importance of each output channel of each convolutional layer in each residual block, to obtain a reconstructed image classification model; the reconstructed image classification model is used to classify the image to be processed.

[0051] By implementing steps 101 to 104 above, dynamic simplification of the model structure and parameters is achieved through joint optimization of convolutional channels and convolutional layer shrinkage. This application innovatively introduces a learnable structural importance evaluation module (channel shrinkage module and layer shrinkage module) to synchronously generate importance coefficients for each convolutional layer and its internal channels during model training. Gradient backpropagation is used to automatically learn the contribution of features at different levels, and a threshold criterion is combined to dynamically prune redundant layers and inefficient channels. Through coupled analysis of inter-layer and inter-channel importance, fine-grained compression and coarse-grained structural optimization are synergistically achieved, effectively preserving the ability to transfer key features during network parameter compression.

[0052] In another exemplary embodiment of this application, such as Figure 4 As shown, the residual block of the channel shrinking module and the layer shrinking module specifically includes: a first convolutional layer, a first normalization layer, a channel shrinking module, a first activation layer, a second convolutional layer, a second normalization layer, a layer shrinking module, an addition layer, and a second activation layer connected in sequence; the input of the addition layer is also connected to the output of the first normalization layer.

[0053] (1) The residual blocks further extract image features through the first convolutional layer, and change the feature map size and number of channels. In all residual blocks (8 residual blocks), the kernel size of the first convolutional layer is 3, the padding is 1, the stride is 1, 1, 2, 1, 2, 1, 2, 1, and the number of output channels is 64, 64, 128, 128, 256, 256, 512, 512. The feature map obtained through the first convolutional layer is sent to the first normalization layer for processing, and the normalized feature map is sent to the subsequent structure for processing.

[0054] (2) The normalized feature map obtained in step (1) is fed into the channel shrinking module for processing. The channel shrinking module uses the soft thresholding method and its internal structure to calculate an adaptive threshold, dynamically selecting data with a smaller absolute value between the threshold and the input feature value for output. At the same time, the channel shrinking module obtains the importance value of each channel in the first convolutional layer for network reconstruction. The feature map obtained by the channel shrinking module is fed into the first activation layer for processing. The first activation layer uses the ReLU activation function to introduce nonlinear characteristics into the data. The obtained data will be fed into subsequent structures for processing.

[0055] (3) The nonlinearly processed features obtained in step (2) are fed into the second convolutional layer for further processing to extract image features. In all residual blocks (8 residual blocks), the structure of the second convolutional layer is the same except for the output channels: the kernel size is 3, the stride is 1, the padding is 1, and the number of output channels is 64, 64, 128, 128, 256, 256, 512, 512 respectively. The features obtained through the second convolutional layer are fed into the second normalization layer for processing, and the obtained normalized feature map is fed into the subsequent structure for processing.

[0056] (4) The normalized feature map obtained in the previous step (3) is sent to the layer shrinkage module for processing. The layer shrinkage module uses the soft thresholding method to calculate the adaptive threshold using its internal structure, and dynamically filters out the data with smaller absolute values ​​between the threshold and the input feature values ​​for output. The filtered data will be sent to the subsequent structure for processing. At the same time, the importance values ​​of each channel in the second convolutional layer and the importance value of the current residual block are obtained for network reconstruction.

[0057] (5) Construct a residual connection between the feature data obtained in the previous step (4) and the normalized features output by the first normalization layer, and process the summation result using the second activation layer. The resulting feature map will be used as the final output of the current residual block and sent to the subsequent network for processing.

[0058] Taking the first residual block in the PruneNet18 network as an example, the specific data flow of a single residual block during training is as follows:

[0059] 1) First, the image features extracted by the network... Input the first convolutional layer (kernel 3, padding 1, stride 1, output channels 64) to obtain the feature map. Furthermore, features are obtained through the first normalization layer. This feature will serve as the input to the channel shrinking module and will form a residual connection with the output of the layer shrinking module.

[0060] 2) Channel shrinking module As input, data exceeding a threshold is filtered out using a soft threshold method to obtain features. and the importance values ​​of each channel in the first convolutional layer Where Ch-11 is 64, representing the number of output channels of the first convolutional layer, and the feature... It will be used as the input to the second convolutional layer.

[0061] 3) Utilizing the first activation layer as a feature Introducing nonlinear characteristics, we obtain Furthermore, The data is fed into a second convolutional layer (3 kernels, 1 stride, 1 padding, 64 output channels) for feature extraction. This feature will be used as input to the second normalization layer;

[0062] 4) Utilize the second normalization layer to After normalization, we get Furthermore, the layer shrinkage module will use a soft thresholding method to filter out... Data above a threshold are used to obtain features. The importance value θ1 of the first residual block and the importance values ​​of each channel in the second convolutional layer Where Ch-12 is 64, representing the number of output channels of the second convolutional layer; features This will be compared with the output of the first normalized layer. Construct residual connections.

[0063] 5) Features and Construct residual connections to obtain features and Furthermore, using the second activation layer as F′ (1) By introducing nonlinear characteristics, we can obtain the features. F (1) This will be used as the final output of the first residual block and fed into the subsequent network for processing.

[0064] In another exemplary embodiment of this application, such as Figure 5 As shown, the channel shrinking module specifically includes: a first absolute value layer, a first average pooling layer, a first max pooling layer, a first flattening and dimensionality reduction layer, a second flattening and dimensionality reduction layer, a first multi-value mapping module, a first multiplication layer, a second absolute value layer, a first dimensionality increase layer, and a first data selection layer.

[0065] The input of the first absolute value layer is connected to the output of the first normalization layer and the input of the first data selection layer in the residual block. The output of the first absolute value layer is connected to the input of the first average pooling layer and the first max pooling layer, respectively. The output of the first average pooling layer is connected to the input of the first multiplication layer through the first flattening and dimensionality reduction layer. The output of the first max pooling layer is connected to the input of the first multiplication layer through the second flattening and dimensionality reduction layer and the first multi-value mapping module. The output of the first multiplication layer is connected to the input of the first data selection layer through the second absolute value layer and the first dimensionality increase layer, respectively. The output of the first data selection layer is connected to the input of the first activation layer in the residual block.

[0066] The first multi-value mapping module is used to obtain the importance weight of each output channel in the first convolutional layer of the residual block; the importance weight of each output channel in the first convolutional layer is averaged in the batch dimension to obtain the importance value of each output channel in the first convolutional layer.

[0067] The first multiplication layer is used to perform a dot product operation on the output of the first flattening and dimensionality reduction layer and the output of the first multi-value mapping module to obtain the first adaptive threshold.

[0068] The first data selection layer is used to filter the input features of the channel shrinking module using the first preprocessed adaptive threshold, and to output the data with the smaller absolute value between the input feature and the first preprocessed adaptive threshold; the first preprocessed adaptive threshold refers to the threshold after taking the absolute value and increasing the dimension of the first adaptive threshold.

[0069] During training, the specific data flow of the channel shrinking module is as follows:

[0070] First, the output features of the first normalization layer Take absolute value It will be sent to the subsequent structure for processing.

[0071] Second, The sample is fed into the first average pooling layer for processing to obtain... Furthermore, by flattening it, its dimensionality is reduced to... Features after dimensionality reduction Perform a dot product operation with the output of the first multi-value mapping module.

[0072] Third, The sample is fed into the first max pooling layer for processing to obtain... Furthermore, by flattening it, its dimensionality is reduced to... Features after dimensionality reduction It will be sent to the first multi-value mapping module for processing.

[0073] Fourth, the features obtained in step three The data is fed into the first multi-value mapping module to obtain the importance weights of each channel in the first convolutional layer of the residual block. Furthermore, regarding η′ 11 Calculating the mean along the batch dimension will yield the importance values ​​of each channel in the first convolutional layer of the residual block. Used for subsequent network reconstruction; in addition, η′ 11 and Perform dot product.

[0074] Fifth, take the result obtained in step two. The η′ obtained in step four11 Performing the dot product operation, we get Furthermore, Θ′ c Take the absolute value |Θ′ c Then, the dimensionality is increased to obtain the final adaptive threshold.

[0075] Sixth, use the threshold Θ obtained in step five. c Features received by the channel shrinking module Filter and select With Θ c Output the data with smaller absolute values ​​to obtain the features. This is the final output of the channel shrinking module.

[0076] In another exemplary embodiment of this application, such as Figure 6 As shown, the layer shrinkage module specifically includes: a third absolute value layer, a second average pooling layer, a second max pooling layer, a third flattening and dimensionality reduction layer, a fourth flattening and dimensionality reduction layer, a second multi-value mapping module, a single-value mapping module, a second multiplication layer, a third multiplication layer, a fourth absolute value layer, a second dimensionality increase layer, and a second data selection layer.

[0077] The input of the third absolute value layer is connected to the output of the second normalization layer and the input of the second data selection layer in the residual block. The output of the third absolute value layer is connected to the input of the second average pooling layer and the second max pooling layer, respectively. The output of the second average pooling layer is connected to the input of the third multiplication layer via the third flattening and dimensionality reduction layer. The output of the second max pooling layer is connected to the input of the single-value mapping module and the second multi-value mapping module via the fourth flattening and dimensionality reduction layer, respectively. The outputs of the single-value mapping module and the second multi-value mapping module are both connected to the input of the second multiplication layer. The output of the second multiplication layer is connected to the input of the third multiplication layer. The output of the third multiplication layer is connected to the input of the second data selection layer via the fourth absolute value layer and the second dimensionality increase layer, respectively. The output of the second data selection layer is connected to the input of the addition layer in the residual block.

[0078] The second multi-value mapping module is used to obtain the importance weight of each output channel in the second convolutional layer of the residual block; the importance value of each output channel in the second convolutional layer is obtained by averaging the importance weights of each output channel in the batch dimension.

[0079] The single-value mapping module is used to obtain the importance weight of the residual block; the importance value of the residual block is obtained by averaging the importance weight of the residual block over the batch dimension.

[0080] The second multiplication layer is used to perform dot product operations on the outputs of the single-value mapping module and the second multi-value mapping module to obtain the feature map channel weights of the corresponding residual block.

[0081] The third multiplication layer is used to perform dot product operations on the outputs of the second multiplication layer and the third flattening and dimensionality reduction layer to obtain the second adaptive threshold.

[0082] The second data selection layer is used to filter the input features of the shrinkage module of the layer using the second preprocessed adaptive threshold, and to select the data with smaller absolute values ​​between the input features and the second preprocessed adaptive threshold for output; the second preprocessed adaptive threshold refers to the threshold after taking the absolute value and increasing the dimensionality of the second adaptive threshold.

[0083] During training, the specific data flow of the layer shrinkage module is as follows:

[0084] First, the output features of the second normalization layer Take absolute value It will be sent to the subsequent structure for processing.

[0085] Second, The sample is fed into a second average pooling layer for processing to obtain... Furthermore, by flattening it, its dimensionality is reduced to... Features after dimensionality reduction It will be used in subsequent processing.

[0086] Third, The sample is fed into the second max pooling layer for processing to obtain... Furthermore, by flattening it, its dimensionality is reduced to... Features after dimensionality reduction The data will be sent to the single-value mapping module and the second multi-value mapping module for processing, respectively.

[0087] Fourth, the features obtained in step three The data is fed into the second multi-value mapping module to obtain the importance weights of each channel in the second convolutional layer of the residual block. Furthermore, regarding η′ 12 Calculating the mean along the batch dimension will yield the importance values ​​of each channel in the second convolutional layer of the residual block. Used for subsequent network reconstruction; in addition, η′ 12 Perform a dot product operation with the output θ1′ of the single-value mapping module.

[0088] Fifth, the features obtained in step three The data is fed into the single-value mapping module to obtain the importance weight of the corresponding residual block. Furthermore, by taking the mean of θ1′ over the batch dimension, the importance value of the residual block to which it belongs can be obtained. Used for subsequent network reconstruction; in addition, θ1′ will be compared with the output η′ of the second multi-valued mapping module. 12 Perform dot product.

[0089] Sixth, the output θ1′ of the single-value mapping module and the output η′ of the second multi-value mapping module are... 12 Perform a dot product operation to obtain the feature map channel weights of the residual block.

[0090] Seventh, combine the feature map channel weights ω obtained in step six with the feature map channel weights ω obtained in step six. Performing the dot product operation, we get Furthermore, Θ′ a Take the absolute value |Θ′ a Then, the dimensionality is increased to obtain the final adaptive threshold.

[0091] Eighth, utilize the adaptive threshold Θ obtained in step seven. a Features received by the layer shrinking module Filter and select and Θ a Output the data with smaller absolute values ​​to obtain the features. This is the final output of the layer shrinking module.

[0092] In another exemplary embodiment of this application, such as Figure 7 As shown, the first multi-value mapping module specifically includes: a first linear layer, a third normalization layer, a third activation layer, a second linear layer, a first sigmoid layer, an averaging layer, and an extension layer connected in sequence.

[0093] The second multi-value mapping module in the layer shrinking module has the same structure as the first multi-value mapping module, but it processes different data.

[0094] Taking the second multi-value mapping module in the layer shrinkage module as an example, the data flow of the second multi-value mapping module during training is as follows:

[0095] First, utilize the first linear layer to analyze channel features. Perform mapping processing to obtain features Furthermore, the third normalization layer is used to... After normalization, we get Furthermore, utilizing the third activation layer as Introducing nonlinear characteristics, we obtain

[0096] Second, using the second linear layer Further mapping yields Furthermore, the sigmoid() function is used to... By processing and introducing nonlinear characteristics, the features are obtained.

[0097] Third, regarding features Calculate the average over the batch dimension to obtain Furthermore, Expand to This facilitates subsequent joint processing of data output from other modules; This is the final output of the multi-valued mapping module. Furthermore, for η′... 12 After averaging over the batch dimension, we get This represents the importance value of each channel in the corresponding convolutional layer.

[0098] In another exemplary embodiment of this application, such as Figure 8 As shown, the single-value mapping module specifically includes: a third linear layer, a fourth normalization layer, a fourth activation layer, a fourth linear layer, and a second sigmoid layer connected in sequence.

[0099] During training, the data flow of the single-value mapping module is as follows:

[0100] First, utilize the third linear layer to analyze channel features. Perform mapping processing to obtain features Furthermore, the fourth normalization layer is used to... After normalization, we get Furthermore, utilizing the fourth activation layer as Introducing nonlinear characteristics, we obtain

[0101] Second, using the fourth linear layer to Mapped to And further utilize the sigmoid() function to By processing and introducing nonlinear characteristics, we obtain... θ1′ will be the final output of the layer shrinkage module. Furthermore, averaging θ1′ over the batch dimension yields a constant θ, which represents the importance value of the corresponding residual block.

[0102] In another exemplary embodiment of this application, step 103, training the improved image classification model using the target image training set, specifically includes:

[0103] (1) Construct a loss function; the loss function includes an L1 loss term and a cross-entropy loss term; the L1 loss term is constructed based on the importance of each residual block and the importance of each output channel of each convolutional layer in each residual block; the cross-entropy loss term is constructed based on the classification prediction result and the true classification label.

[0104] The specific expression for the loss function is as follows:

[0105]

[0106] Where, ω θ ω represents the penalty coefficient for the loss of the importance value of the residual block. η The penalty coefficient is the sum of the channel importance values ​​for the loss. These penalty coefficients are preset values ​​and do not participate in backpropagation. ce(P,G) represents the cross-entropy loss constructed using the classification prediction result P and the true classification label G. L1 loss can promote network sparsity while reducing the performance loss of the reconstructed network.

[0107] (2) Based on the loss function, the improved image classification model is trained using the target image training set. For example... Figure 9 As shown.

[0108] by Figure 3 Taking the PruneNet18 network structure shown as an example, this illustrates the training data direction of an image classification model with added channel shrinkage and layer shrinkage modules:

[0109] 1) Using the initial convolutional layer, the input image is initially extracted. The characteristics, among which Furthermore, the obtained feature map The data is fed into the initial normalization layer to obtain a normalized feature map. Regularizing the data reduces the risk of overfitting and improves the model's generalization ability. Finally, The feature map is obtained by feeding it into the activation layer at the beginning and introducing nonlinear characteristics. F (0) It will be sent to the subsequent network for processing.

[0110] 2) The feature map F obtained in step 1) (0) The data is fed into eight cascaded residual blocks for processing, where the output feature map of the i-th residual block is F. (i) The resulting feature maps are as follows:

[0111] At the same time, each residual block will output its importance value θ. i, i∈[1,8], and the importance values ​​of each channel in the convolutional layer. The resulting channel importance values ​​are as follows: The obtained structural importance values ​​will be added to the block importance list and the channel importance list respectively for use in subsequent reconstruction. j represents the j-th convolutional layer in the residual block, j∈[1,2], ij represents the j-th convolutional layer in the i-th residual block, and Ch-ij represents the number of output channels of the j-th convolutional layer in the i-th residual block.

[0112] The first multi-value mapping module in the channel shrinking module is used to obtain the convolutional channel importance information calculated from a batch of images. Furthermore, the average value is calculated over the batch dimension to obtain the importance value of each channel in the first convolutional layer of the corresponding residual block.

[0113] The second multi-value mapping module in the layer shrinkage module is used to obtain the convolutional channel importance information calculated from a batch of images. Furthermore, the average value is calculated over the batch dimension to obtain the importance value of each channel in the second convolutional layer of the corresponding residual block.

[0114] Using the single-value mapping module in the layer shrinkage module, the importance information of residual blocks calculated from a batch of images is obtained. Furthermore, the average value is calculated over the batch dimension to obtain the importance value of the residual block to which it belongs.

[0115] The importance value of each convolutional layer channel Store the channel importance list. Set the importance value θ for each residual block. i Store the block importance list; the trace importance list and the block importance list will be used for subsequent steps such as loss value calculation and network reconstruction.

[0116] As an example, this embodiment can pre-set the number of iterations for outputting the importance values ​​of each structure, for example, recording the importance values ​​of each structure every certain number of iterations.

[0117] During the training process, please refer to the specific process described above for the data flow direction of each residual block, the channel shrinking module, and the layer shrinking module; it will not be repeated here.

[0118] 3) The obtained feature F (8) The data is fed into the initial pooling layer and subjected to adaptive average pooling to obtain global features. And further reduce the dimensionality to a global feature vector. Finally, it is mapped to the predicted classification result P by the initial linear layer.

[0119] 4) Construct the L1 loss value using the structural importance information obtained in step 2), and construct the cross-entropy loss value using the prediction result P and the true label G. Sum the two losses as the total loss value.

[0120] 5) Optimize network parameters by backpropagation based on the total loss obtained in step 4).

[0121] 6) Save the network parameters and the obtained importance information of each structure. The main training process ends here.

[0122] In another exemplary embodiment of this application, step 104 involves reconstructing the initial image classification model based on the importance of each residual block corresponding to multiple preset iteration numbers and the importance of each output channel of each convolutional layer in each residual block, to obtain a reconstructed image classification model. Specifically, this includes:

[0123] (1) Determine the image classification accuracy of the trained image classification model corresponding to multiple preset iteration numbers;

[0124] (2) Based on the importance of each residual block of the trained image classification model corresponding to the optimal image classification accuracy and the importance of each output channel of each convolutional layer in each residual block;

[0125] (3) Based on the residual block threshold and channel threshold, each residual block, each convolutional layer in the residual block, and each output channel in the convolutional layer are selected in the initial image classification model. The selected residual blocks and convolutional layer output channels are assigned corresponding weights to obtain the reconstructed image classification model.

[0126] After training, the network is reconstructed based on the channel importance list and block importance list obtained during training. The main process for reconstructing the image classification model is as follows: First, based on the original network structure, an initial feature extraction structure is added. Then, based on the structural importance information, relevant structures with importance values ​​greater than a threshold are added to the network. Finally, the terminal structure that implements the classification decision is added. Simultaneously, corresponding parameters are extracted from the original parameter file based on the added structures and used as parameters for network reconstruction. This completes the network reconstruction process. The main process for reconstructing the image classification model is as follows: Figure 10 As shown.

[0127] SC1: Add the starting structure and corresponding parameters.

[0128] Taking the PruneNet18 network as an example, the schematic diagram of the added starting network structure is as follows: Figure 11As shown, the initial network structure specifically includes three structures: an initial convolutional layer (7×7 kernel, stride 2, padding 1, input channels 3, output channels 64), an initial normalization layer, and an initial activation layer. These structures are used for preliminary image feature extraction. After adding these structures, relevant portions are extracted from the original network's weight file and saved to a new weight file.

[0129] SC2: Add the corresponding structure and parameters according to the importance list.

[0130] The block importance list obtained during training is traversed module by module. When the block importance value is greater than the preset block threshold, the positions of the channel importance values ​​in the channel importance list that are greater than the channel threshold are stored in the structure position variables indx1 and indx2 according to the order in which the corresponding convolutional layers are located in the residual modules. Since the structure importance values ​​are stored sequentially during training, the positions of each value in the table are the positions of the corresponding structures in the network.

[0131] Meanwhile, based on the obtained structural position variables indx1 and indx2, the weight parameters of important structures are selected from the weight file and saved as a new weight file; the new weight file is loaded onto the new structure, thus obtaining the compressed new network CompressedNet18.

[0132] The specific process of reconstructing the network and adding basic structures is as follows: Figure 12 As shown. Based on the number of elements in variable indx2, i.e., whether the variable length is 0, the corresponding basic structures are added to the new network. In the reconstruction process, the process only proceeds to step SC2 when the residual block importance value is greater than a preset threshold. When the residual block importance value is not greater than the preset threshold, it means that no structure in the residual block needs to be retained. After entering step SC2, it is determined whether the length of indx1 and indx2 is 0 to determine the convolutional layers and their output channels that need to be retained in the residual block. When the length of indx1 is 0 and the length of indx2 is not 0, it means that only the second convolutional layer and its subsequent related structures in the residual block are retained, i.e., the following structures are added: Figure 13 The basic structure shown is incorporated into the new network. In this case, basic structure one corresponds to the second convolutional layer and related processing structure of the corresponding residual block in the original network. When the length of indx1 is not 0 and the length of indx2 is 0, it means that only the structure before the second convolutional layer in the corresponding residual block is retained, and the results after the second convolutional layer are not retained. This means that the following structures are added: Figure 13 The basic structure shown is incorporated into the new network. In this case, the basic structure corresponds to the first convolutional layer and related processing structures of the residual block in the original network. When the lengths of indx1 and indx2 are not 0, the following will be added: Figure 14In the new network shown, basic structure two corresponds to the two convolutional layers and related processing structures of the residual block in the original network.

[0133] SC3: Add the end structure and corresponding parameters.

[0134] like Figure 15 As shown, the relevant structures of the terminal network are added, namely the terminal pooling layer, the dimension reduction layer, and the terminal linear layer (corresponding to...). Figure 15 The fully connected layer in the network further maps the extracted features into classification confidence information, and the fully connected layer makes the final classification decision as the final output of the network; at the same time, the corresponding part of the original network weights is saved to a new weight file.

[0135] In this application, during image classification model training, channel shrinking and layer shrinking modules are used to obtain the importance information of each layer structure and each channel within a layer, ultimately resulting in a block importance list and a channel importance list. By comparing these lists with corresponding threshold values, the importance of each structure within the model to the task is determined. Furthermore, based on the block importance list and channel importance list, structures with high influence are retained, and the image classification model is reconstructed. This compression method effectively reduces the structural complexity of the image classification model, decreases the number of model parameters, significantly reduces the computational load of the network, and simultaneously ensures that the classification performance of the compressed model remains essentially unchanged.

[0136] In another exemplary embodiment of this application, an application method for an image classification model is provided, such as... Figure 16 As shown, it includes:

[0137] S1: Obtain the image to be processed.

[0138] S2: Input the image to be processed into the image classification model to obtain the image classification result; the image classification model is a reconstructed image classification model based on the above-mentioned image classification model compression method that combines convolutional channels and convolutional layer shrinkage.

[0139] This application also provides an application scenario that utilizes the aforementioned image classification model compression method combining convolutional channels and convolutional layer shrinkage, as well as the image classification model application method. Specifically, it can be applied in autonomous driving scenarios. This scenario includes a training and compression stage for the image classification model and an image classification stage. The training and compression stage trains the initial image classification model and prunes redundant and inefficient layers using the importance of convolutional layers and convolutional layer channels during training, reconstructing a network structure that has a high impact on the image classification accuracy. The image classification stage classifies objects in images collected in actual traffic scenarios based on the reconstructed image classification model, and adjusts the autonomous driving strategy based on the image classification results. The image classification model compression method combining convolutional channels and convolutional layer shrinkage provided in this embodiment belongs to the image classification model training and compression stage, while the image classification model application method belongs to the image classification stage.

[0140] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the above-described image classification model compression method combining convolutional channels and convolutional layer shrinkage, or the above-described image classification model application method.

[0141] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0142] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image classification model compression method combining convolution channel and convolution layer shrinkage, characterized in that, The application relates to an improved image classification model. The initial image classification model comprises a fixed start-end network structure, a fixed end-end network structure and a plurality of residual blocks arranged between the start-end network structure and the end-end network structure; the residual blocks comprise two convolution layers; A channel contraction module is connected in series after the first convolution layer of each residual block, and a layer contraction module is connected in series after the second convolution layer, so as to obtain an improved image classification model; the channel contraction module is used for calculating a first adaptive threshold value by using a soft threshold method, dynamically screening data with smaller absolute values from the first adaptive threshold value and input feature values of the channel contraction module for output, and obtaining the importance of each output channel in the first convolution layer; the layer contraction module is used for calculating a second adaptive threshold value by using a soft threshold method, dynamically screening data with smaller absolute values from the second adaptive threshold value and input feature values of the layer contraction module for output, and obtaining the importance of each output channel in the second convolution layer and the importance of the residual block; The improved image classification model is trained by using a target image training set, and in the training process, the importance of each residual block and the importance of each output channel in each convolution layer in each residual block corresponding to a plurality of preset iteration times are recorded; The initial image classification model is reconstructed according to the importance of each residual block and the importance of each output channel in each convolution layer in each residual block corresponding to a plurality of preset iteration times, so as to obtain a reconstructed image classification model; the reconstructed image classification model is used for classifying a to-be-processed image. The residual block with the channel contraction module and the layer contraction module connected in series comprises a first convolution layer, a first normalization layer, a channel contraction module, a first activation layer, a second convolution layer, a second normalization layer, a layer contraction module, an addition layer and a second activation layer connected in sequence; the input end of the addition layer is also connected to the output end of the first normalization layer.

2. The method of claim 1, wherein the method further comprises: The channel contraction module comprises a first absolute value layer, a first average pooling layer, a first maximum pooling layer, a first flattening and dimension reduction layer, a second flattening and dimension reduction layer, a first multi-value mapping module, a first multiplication layer, a second absolute value layer, a first dimension increasing layer and a first data selection layer.

3. The method of claim 1 or 2, wherein the method is characterized by, The input end of the first absolute value layer is connected to the output of the first normalization layer and the input end of the first data selection layer in the residual block; the output end of the first absolute value layer is connected to the input end of the first average pooling layer and the first maximum pooling layer, the output end of the first average pooling layer is connected to the input end of the first multiplication layer through the first flattening and dimension reduction layer; the output end of the first maximum pooling layer is connected to the input end of the first multiplication layer through the second flattening and dimension reduction layer and the first multi-value mapping module in sequence, the output end of the first multiplication layer is connected to the input end of the first data selection layer through the second absolute value layer and the first dimension increasing layer in sequence, and the output end of the first data selection layer is connected to the input end of the first activation layer in the residual block. ​ The first multi-value mapping module is configured to obtain importance weights of each output channel in the first convolutional layer in the residual block; and the importance weights of each output channel in the first convolutional layer are averaged in the batch dimension to obtain importance values of each output channel in the first convolutional layer. The first multiplication layer is configured to perform point multiplication operation on the output of the first flattening and dimension reduction layer and the output of the first multi-value mapping module to obtain the first adaptive threshold value. The first data selection layer is configured to filter the input features of the channel contraction module by using the first preprocessed adaptive threshold value, and output data with smaller absolute values in the input features and the first preprocessed adaptive threshold value; the first preprocessed adaptive threshold value refers to a threshold value obtained by taking absolute value and dimension increasing of the first adaptive threshold value.

4. The method of claim 3, wherein the method further comprises: The layer contraction module specifically comprises: a third absolute value taking layer, a second average pooling layer, a second maximum pooling layer, a third flattening and dimension reduction layer, a fourth flattening and dimension reduction layer, a second multi-value mapping module, a single-value mapping module, a second multiplication layer, a third multiplication layer, a fourth absolute value taking layer, a second dimension increasing layer, and a second data selection layer. The output end of the third absolute value taking layer is connected to the output of the second normalization layer in the residual block and the input end of the second data selection layer; the output end of the third absolute value taking layer is connected to the input end of the second average pooling layer and the input end of the second maximum pooling layer; the output end of the second average pooling layer is connected to the input end of the third multiplication layer through the third flattening and dimension reduction layer; the output end of the second maximum pooling layer is sequentially connected to the input end of the single-value mapping module and the input end of the second multi-value mapping module through the fourth flattening and dimension reduction layer; the output end of the single-value mapping module and the output end of the second multi-value mapping module are connected to the input end of the second multiplication layer; the output end of the second multiplication layer is connected to the input end of the third multiplication layer; the output end of the third multiplication layer is sequentially connected to the input end of the second data selection layer through the fourth absolute value taking layer and the second dimension increasing layer; and the output end of the second data selection layer is connected to the input end of the addition layer in the residual block. The second multi-value mapping module is configured to obtain importance weights of each output channel in the second convolutional layer in the residual block; and the importance weights of each output channel in the second convolutional layer are averaged in the batch dimension to obtain importance values of each output channel in the second convolutional layer. The single-value mapping module is configured to obtain importance weights of the residual block; and the importance weights of the residual block are averaged in the batch dimension to obtain importance values of the residual block. The second multiplication layer is configured to perform point multiplication operation on the outputs of the single-value mapping module and the second multi-value mapping module to obtain feature map channel weights of the residual block. The third multiplication layer is configured to perform point multiplication operation on the outputs of the second multiplication layer and the third flattening and dimension reduction layer to obtain a second adaptive threshold value. The second data selection layer is configured to filter the input features of the layer contraction module by using the second preprocessed adaptive threshold value, and output data with smaller absolute values in the input features and the second preprocessed adaptive threshold value; the second preprocessed adaptive threshold value refers to a threshold value obtained by taking absolute value and dimension increasing of the second adaptive threshold value.

5. The method of claim 4, wherein the method further comprises: The first multi-value mapping module specifically comprises a first linear layer, a third normalization layer, a third activation layer, a second linear layer, a first sigmoid layer, an average taking layer and an expansion layer connected in sequence.

6. The method of claim 4, wherein the method further comprises: The single-value mapping module specifically comprises a third linear layer, a fourth normalization layer, a fourth activation layer, a fourth linear layer and a second sigmoid layer connected in sequence.

7. The method of claim 1, wherein the method further comprises: The improved image classification model is trained by using a target image training set, and specifically includes: A loss function is constructed; the loss function includes an L1 loss term and a cross-entropy loss term; the L1 loss term is constructed according to the importance of each residual block and the importance of each output channel of each convolution layer in each residual block; the cross-entropy loss term is constructed according to the classification prediction result and the real classification label; Based on the loss function, the improved image classification model is trained by using a target image training set.

8. The method of claim 1, wherein the method further comprises: The initial image classification model is reconstructed according to the importance of each residual block and the importance of each output channel of each convolution layer in each residual block corresponding to a plurality of preset iteration numbers, to obtain a reconstructed image classification model, specifically including: The image classification accuracy of the trained image classification model corresponding to a plurality of preset iteration numbers is determined; According to the importance of each residual block and the importance of each output channel of each convolution layer in each residual block of the trained image classification model corresponding to the optimal image classification accuracy; First, according to the residual block threshold, the importance of each residual block in the initial image classification model greater than the residual block threshold is screened, then according to the channel threshold, the importance of the convolution layer output channel of the output channel in the residual block greater than the channel threshold is screened, and finally the screened residual block and convolution layer output channel are assigned corresponding weights, to obtain the reconstructed image classification model.

9. An application method of an image classification model, characterized in that, It includes: An image to be processed is acquired; The image to be processed is input into the image classification model to obtain an image classification result; The image classification model is a reconstructed image classification model obtained by the image classification model compression method combining convolution channel and convolution layer shrinkage according to any one of claims 1 to 8.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by a processor to realize the image classification model compression method combining convolution channel and convolution layer shrinkage according to any one of claims 1 to 8, or the application method of the image classification model according to claim 9. The computer program is executed by a processor to realize the image classification model compression method combining convolution channel and convolution layer shrinkage according to any one of claims 1 to 8, or the application method of the image classification model according to claim 9.

Images