Lightweight image quality assessment method and system based on convolutional neural network pruning
By splitting the convolutional layers of a convolutional neural network into sub-convolutional kernels and decorators, and training and fine-tuning on small datasets, the problem of deploying image quality assessment models on resource-constrained devices is solved, achieving rapid lightweighting and performance improvement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2023-10-24
- Publication Date
- 2026-07-24
AI Technical Summary
Existing image quality assessment models are difficult to deploy on resource-constrained devices, the model training process is complex and computationally expensive, and performance degrades significantly after pruning, especially under small database conditions.
The convolutional layers of the convolutional neural network are split into sub-convolutional kernels and decorators. The decorators are initialized using a hierarchical correlation propagation algorithm, trained on a small image quality assessment dataset, fine-tuned after pruning, and the convolutional layers are merged to restore performance.
We have developed a fast and lightweight image quality assessment model for small datasets. It is faster to compute, requires less storage space, and even has improved performance, making it suitable for resource-constrained devices.
Smart Images
Figure CN117372387B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal media quality assessment technology, specifically to a lightweight image quality assessment method, system, and terminal based on convolutional neural network pruning. Background Technology
[0002] The goal of blind image quality assessment is to evaluate the quality of images without reference images at various stages of image processing, such as image acquisition, compression, and transmission. In recent years, deep neural networks have achieved significant performance in image quality assessment due to their powerful feature representation capabilities.
[0003] Despite the great success these DNN-based models have achieved in terms of performance, model inference is expensive due to the time, power, and memory required by their heavyweight architecture, making them difficult to deploy on resource-constrained devices and requiring significant computational resources during inference.
[0004] Filter pruning is one of the most popular methods in model compression, reducing both parameters and floating-point operations (FLOPs) by removing unimportant convolutional channels. For example, some researchers calculate the L1 norm of each filter weight as a measure of channel importance; others use the scaling factor of the batch normalization (BN) layer following the convolutional layers to measure channel importance. Despite significant differences between these pruning methods, most involve the following steps: evaluating the importance of each filter, pruning unimportant filters, and retraining the entire model to restore its performance. However, image quality assessment is often based on transfer learning, and the model training process is typically complex. For instance, DBCNN pre-trains two modules on a database containing over 80,000 distorted images for distortion classification and on ImageNet for image classification, respectively, before fine-tuning the entire model after training the regressor. Another example is Stair image quality assessment, which uses ResNet-50 as its backbone, initialized by pre-training on ImageNet and then simultaneously trained on multiple databases. The complexity of the training process for no-reference image quality assessment models and the large database used in the pre-training phase make retraining after pruning resource-intensive and slow, especially when using traditional iterative pruning processes.
[0005] Similar to image quality assessment, other transfer learning-based models also face the problem of excessive computational resource consumption during pruning. Some research on transfer learning pruning has emerged, but pruning is difficult without the large databases used in pre-training. Other studies train scaling factors in Batch Normalization layers after convolutional layers to measure the importance of each convolutional channel; however, this method causes varying degrees of damage to different parts of the network, making retraining after pruning more difficult. Furthermore, TransTailor uses scaling factors in Batch Normalization to identify the most important channels, which some parts of the image quality assessment model may not have equipped with. Using these methods to prune image quality assessment methods results in a significant performance degradation, or requires substantial computational resources to recover the performance.
[0006] Therefore, it is necessary to develop a lightweight image quality assessment technique to solve the above problems. Summary of the Invention
[0007] To overcome the shortcomings of the prior art, the purpose of this invention is to provide a lightweight image quality assessment method, system, and terminal based on convolutional neural network pruning, which can complete the image quality assessment task with faster computing speed and less storage space, while ensuring image quality assessment performance.
[0008] The first objective of this invention is to provide a lightweight image quality assessment method based on convolutional neural network pruning, comprising:
[0009] Obtain the dataset to be evaluated;
[0010] Select a convolutional neural network structure based on the dataset to be evaluated, and train the convolutional neural network.
[0011] The trained convolutional neural network is pruned to obtain an image quality assessment model;
[0012] The image quality evaluation model is input into the dataset to be evaluated to obtain the image quality evaluation result;
[0013] Wherein: the pruning operation on the trained convolutional neural network to obtain the image quality assessment model includes:
[0014] The convolutional layers in the convolutional neural network are split into two trainable parts: sub-convolutional kernels and decorators, and the decorators are initialized using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer.
[0015] The decorator is trained using the dataset to be evaluated;
[0016] Set the mask according to the decorative element and complete the pruning;
[0017] The pruned convolutional neural network is fine-tuned using the dataset to be evaluated to obtain an image quality evaluation model.
[0018] Optionally, the step of selecting a convolutional neural network structure based on the dataset to be evaluated and training the convolutional neural network includes:
[0019] Based on the dataset to be evaluated, a convolutional neural network structure is selected, and image classification is used as the source task to complete pre-training on a public dataset.
[0020] The pre-trained convolutional neural network is further trained on the dataset to be evaluated to complete transfer learning. Specifically, the feature extraction part of the pre-trained convolutional neural network is connected to a new regressor, and transfer learning is performed on image quality evaluation to improve the performance of image quality evaluation by utilizing the semantic knowledge in the feature extractor.
[0021] Optionally, training the decorator using the dataset to be evaluated includes:
[0022] During the pruning phase, each part of the convolutional neural network, except for the decorator, is frozen;
[0023] The decorator is trained using the target image quality assessment database, i.e., the dataset to be evaluated, to better adapt to the image quality assessment task;
[0024] The trained decorator is converted into a global importance factor for each channel using Taylor expansion, and a pruning mask is set under the guidance of the importance factor.
[0025] The target image quality evaluation database, i.e. the dataset to be evaluated, described above in this invention can be a small-scale dataset, thereby enabling faster computation and requiring less storage space.
[0026] Optionally, the step of setting a mask and completing the cut based on the decorator includes:
[0027] The importance of each channel is measured using the global importance factor, the mask of the least important channel is set to 0, and the masks of the remaining channels are set to 1; during the pruning phase, the output of each channel is multiplied by the mask to simulate pruning.
[0028] Optionally, the pruned convolutional neural network is fine-tuned using the dataset to be evaluated to obtain an image quality assessment model.
[0029] The convolutional layers in the convolutional neural network are merged, and the entire convolutional neural network except for the decorator is retrained using the dataset to be evaluated to mitigate the performance loss caused by pruning; after retraining, the convolutional layers are split again using the unchanged decorator.
[0030] The pruning operation is iterated until the compression ratio reaches the target or the performance is lower than a predefined threshold, thus obtaining the image quality evaluation model.
[0031] Optionally, merging the convolutional layers in the convolutional neural network is to restore the convolutional layers to their normal state, wherein the "merge" operation is defined as the product between the sub-convolutional kernel and its corresponding decorator.
[0032] Optionally, the convolutional layers in the convolutional neural network are split into sub-convolutional kernels and decorators, wherein: the splitting of the convolutional layers is achieved by factoring the convolutional kernels;
[0033] The original convolution kernel W for each channel ij Divided into sub-convolution kernels and Decorator α ij And they are all trainable:
[0034]
[0035] Where i and j represent the i-th layer and the j-th channel of the current layer in the image quality assessment model, respectively; decorators are used to measure the importance of each channel, and the least important channels are pruned under the guidance of these decorators; in the split state, the input feature map F from the previous layer i-1 With sub-convolution kernel W i * Perform as The convolution operation is then performed with the decorator α in the i-th layer. i Multiplication:
[0036]
[0037] Thus, the output feature map F i The results of the operation are the same before and after the split, and the split operation does not cause any additional errors in the reasoning results.
[0038] Optionally, the decorator is initialized using a hierarchical dependency propagation algorithm, wherein:
[0039] The lost image x is fed into the convolutional neural network for inference, and the activation map of each convolutional layer is collected.
[0040] The output f(x) of the convolutional neural network is propagated backward through the network according to the LRP propagation rule, and all activation maps obtain their correlation with the output of the image quality assessment model.
[0041] The correlations of activations belonging to the same convolutional channel are summed to initialize the correlation of the decorator for that channel.
[0042] The second objective of this invention is to provide a lightweight image quality assessment system based on convolutional neural network pruning, comprising:
[0043] Image acquisition module: Acquires the dataset to be evaluated;
[0044] Model pre-training module: Selects a convolutional neural network structure based on the dataset to be evaluated, and trains the convolutional neural network;
[0045] Model pruning module: Prunes the trained convolutional neural network to obtain an image quality assessment model;
[0046] Evaluation module: Input the dataset to be evaluated into the image quality evaluation model to obtain the image quality evaluation result;
[0047] Wherein: the model pruning module includes:
[0048] The convolutional layer in the convolutional neural network is split into two trainable parts: a sub-convolutional kernel and a decorator, and the decorator is initialized using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer.
[0049] The decorator is trained using the dataset to be evaluated;
[0050] Set the mask according to the decorative element and complete the pruning;
[0051] The pruned convolutional neural network is fine-tuned using the dataset to be evaluated to obtain an image quality evaluation model.
[0052] The third objective of this invention is to provide a lightweight image quality assessment terminal based on convolutional neural network pruning, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it performs the aforementioned lightweight image quality assessment method based on convolutional neural network pruning.
[0053] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:
[0054] This invention provides a lightweight image quality assessment method and system based on convolutional neural network pruning. The method involves pre-training on an image quality assessment dataset followed by pruning. The pruned model only needs retraining on a smaller image quality assessment dataset, resulting in a faster and more storage-efficient lightweight image quality assessment model for image evaluation. Dividing the convolutional layer into sub-kernels and decorators allows for more accurate measurement of the importance of convolutional channels, thus reducing the requirements for retraining conditions. Experiments on several image quality assessment models demonstrate the effectiveness of the image quality assessment method in this invention.
[0055] Furthermore, because this invention can identify important, useless, and harmful filters, the performance of some image quality assessment models can even be improved by model compression at an appropriate pruning rate. The improved performance includes higher Spearman and Pearson correlation coefficients between the inference results and the true MOS scores of the image quality assessment method. In practical image assessment applications, more accurate image quality assessment results can be obtained with less storage space and shorter running time. Attached Figure Description
[0056] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:
[0057] Figure 1 This is a flowchart of an image quality evaluation method according to an embodiment of the present invention;
[0058] Figure 2 This is a flowchart of an image quality evaluation method according to a preferred embodiment of the present invention;
[0059] Figure 3 This is a schematic diagram of data reasoning during the pruning operation in one embodiment of the present invention. Detailed Implementation
[0060] The embodiments of the present invention are described in detail below: These embodiments are implemented based on the technical solution of the present invention, and provide detailed implementation methods and specific operation processes. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.
[0061] In existing technologies, image quality assessment methods suffer from complex model training. Traditional iterative pruning processes require large databases, making retraining after pruning resource-intensive and slow, resulting in slow implementation speeds and large training data requirements. To avoid using multiple large databases for complex retraining processes and to ensure the generalizability of most image quality assessment models, this invention proposes a resource-efficient and easy-to-use lightweight image quality assessment method. This method uses only a small image quality assessment dataset to complete the entire pruning process while maintaining the model's performance on the dataset used. Furthermore, since pruning can identify important, useless, and harmful filters, the performance of the image quality assessment model can even be improved after compression with an appropriate pruning rate. Thus, a lightweight image quality assessment model with faster computation speed and smaller storage space is obtained while maintaining performance. Using this model for image quality assessment can significantly reduce computational resource consumption and improve processing speed.
[0062] Figure 1 This is a flowchart of an image quality evaluation method according to an embodiment of the present invention. (Refer to...) Figure 1 As shown, the lightweight image quality assessment method based on convolutional neural network pruning in this embodiment includes the following steps:
[0063] S100: Obtain the dataset to be evaluated;
[0064] In this step, the dataset to be evaluated can be a quality assessment database containing target images. Specifically, the dataset to be evaluated consists of images whose perceived quality needs to be assessed and their corresponding average subjective scores. The images can be obtained from social networking sites, online media, and other sources.
[0065] S200: Select a convolutional neural network structure based on the dataset to be evaluated, and train the convolutional neural network;
[0066] In this step, a convolutional neural network (CNN) architecture is selected based on the dataset to be evaluated. This CNN architecture can be chosen from publicly available, high-performing CNN-based image quality assessment models within the existing image quality assessment field. Specifically, the dataset to be evaluated can be evenly divided into a training set and a test set. The existing image quality assessment model is trained using the training set and tested on the test set. The CNN model with the highest test performance is then selected.
[0067] S300: Pruning the trained convolutional neural network to obtain an image quality assessment model;
[0068] This step may include the following:
[0069] S301: The convolutional layer in the convolutional neural network is split into two trainable parts: sub-convolutional kernels and decorators, and the decorators are initialized using the hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer.
[0070] S302: Train the decorator using the dataset to be evaluated; the dataset to be evaluated can be a small dataset of image quality, such as a dataset of about 1000 images.
[0071] S303: Set the mask according to the decorator and complete the pruning;
[0072] S304: Fine-tune the pruned convolutional neural network using an image quality assessment dataset to obtain an image quality assessment model.
[0073] S400: Input the dataset to be evaluated into the image quality evaluation model to obtain the image quality evaluation results;
[0074] In this embodiment, the convolutional neural network model is first pre-trained on an image quality assessment dataset and then pruned. The pruned model only needs to be retrained on a smaller image quality assessment dataset to obtain a lightweight image quality assessment model with faster computation speed and smaller storage space. Specifically, in this embodiment, the convolutional layer is divided into sub-convolutional kernels and decorators, which allows for more accurate measurement of the importance of the convolutional channels, thereby reducing the requirements for retraining conditions. This makes the overall image assessment computation faster and requires less storage space.
[0075] As a preferred embodiment, refer to Figure 2 As shown, the lightweight image quality assessment method based on convolutional neural network pruning provided in this embodiment includes the following steps:
[0076] S1: Select a suitable convolutional neural network structure based on the dataset to be evaluated, and perform pre-training on the ImageNet dataset with image classification as the source task; of course, in other embodiments, pre-training can also be performed on other public datasets.
[0077] S2: Further train the neural network obtained in S1 on the image quality assessment dataset to complete transfer learning;
[0078] S3: The original convolutional layer in the convolutional neural network is divided into two trainable parts: sub-kernels and decorators, which represent the importance of this channel. Specifically, the decorators are initialized using the hierarchical relevance propagation (LRP) algorithm, which measures the contribution of each unit in the model to the network's information flow;
[0079] S4: During the pruning phase, the convolutional layers are split according to S2, and each part of the model is frozen except for the decorator. Only the decorator is trained using the target image quality assessment database (the dataset to be evaluated) to better adapt to the image quality assessment task. Then, the trained decorator is converted into a global importance factor for each channel using Taylor expansion, and the pruning mask is set under the guidance of the importance factor.
[0080] S5: Using the global importance factor obtained in S4, the importance of each channel is measured, and the mask of the least important channel is set to 0, while the masks of the remaining channels are set to 1. During the pruning phase, the output of each channel is multiplied by the mask to simulate pruning.
[0081] S6: Merge the convolutional layers in the S4 network, and retrain the entire model except for the decorator using only the image quality rating database (the dataset to be evaluated) to mitigate the performance loss caused by pruning. After retraining, split the convolutional layers again using the unchanged decorator;
[0082] S7: S4-S6 can be iterated until the compression ratio reaches the target or the performance falls below a predefined threshold. Because a mask is used to simulate pruning during the pruning phase, it is necessary to extract the actual compact model based on the mask to facilitate deployment after pruning. Retraining after model extraction is optional because the extraction operation does not cause significant performance loss.
[0083] Corresponding to the same concept as the lightweight image quality assessment method based on convolutional neural network pruning in the above embodiments, another embodiment of the present invention also provides an image quality assessment system based on convolutional neural networks, such as... Figure 2 As shown, after obtaining the dataset to be evaluated, the system consists of three main modules:
[0084] Model pre-training module: Select a convolutional neural network structure based on the dataset to be evaluated, and train the convolutional neural network; specifically, optionally, select a suitable convolutional neural network structure based on the dataset to be evaluated, use image classification as the source task to complete pre-training on the ImageNet dataset, and then further train the obtained neural network on the image quality evaluation dataset to complete transfer learning.
[0085] Model pruning module: This module prunes the trained convolutional neural network to obtain an image quality assessment model. This is a crucial step in this embodiment. The convolutional layers in the convolutional neural network are split into sub-convolutional kernels and decorators, and the decorators are initialized using a hierarchical correlation propagation algorithm. The decorators are trained using a small-scale image quality assessment dataset. A mask is set based on the decorators, and pruning is completed. The pruned convolutional neural network is then fine-tuned using the image quality assessment dataset to obtain the image quality assessment model.
[0086] Evaluation module: Input the dataset to be evaluated into the image quality evaluation model to obtain the image quality evaluation results.
[0087] In this embodiment, the model pruning module includes a channel importance-aware part. This part divides the original convolutional layer into two trainable parts: sub-convolutional kernels and decorators, which represent the importance of each channel. Specifically, the decorators are initialized using the hierarchical correlation propagation (LRP) algorithm, which measures the contribution of each unit in the model to the network information flow. During the pruning phase, the convolutional layer is split into sub-convolutional kernels and decorators; except for the decorators, each part of the model is frozen. Only the decorators are trained using the target image quality assessment database (the dataset to be evaluated) to better adapt to the image quality assessment task. Then, Taylor expansion is used to convert the trained decorators into global importance factors for each channel, and pruning masks are set under the guidance of these importance factors.
[0088] Furthermore, a global importance factor can be used to measure the importance of each channel, setting the mask of the least important channel to 0 and the masks of the remaining channels to 1. During the pruning phase, the output of each channel is multiplied by the mask to simulate pruning; the convolutional layers in the network are merged, and the entire model, except for the decorator, is retrained using only the image quality assessment database (the dataset to be evaluated) to mitigate the performance loss caused by pruning. After retraining, the convolutional layers are split again using the unchanged decorator; the above steps are iterated until the compression ratio reaches the target or the performance falls below a predefined threshold. Because masks are used to simulate pruning during the pruning phase, it is necessary to extract an actual compact model based on the mask to facilitate post-pruning deployment; the extracted model is then retrained on the image quality assessment dataset.
[0089] The aforementioned channel importance perception part corresponds to S3-S4 in the embodiment of the lightweight image quality assessment method based on convolutional neural network pruning, and the specific implementation technology is the same.
[0090] The image quality assessment method, system, and terminal based on convolutional neural network pruning provided in the above embodiments of the present invention can quickly and effectively obtain a lightweight image quality assessment model with faster computation speed and smaller storage space.
[0091] In a preferred embodiment of the present invention, the image quality assessment model pre-training module may include the following two parts.
[0092] (1) Image classification is used as the source task and trained on the large ImageNet dataset, so that the network has the ability to understand image semantics;
[0093] (2) Connect the feature extraction part of the pre-trained network in S1 with the new regressor, perform transfer learning on image quality evaluation, and improve the performance of image quality evaluation by utilizing the semantic knowledge in the feature extractor.
[0094] like Figure 2 As shown, in a preferred embodiment, in the model pruning module, channel importance awareness (corresponding to...) Figure 2 The channel importance-aware module (in the code) consists of two parts: splitting the convolutional layer and decorator initialization.
[0095] (1) Splitting the convolutional layer is achieved through kernel factorization. The original convolutional kernel W for each channel... ij Divided into sub-convolution kernels and Decorator α ij And they are all trainable:
[0096]
[0097] Here, i and j represent the i-th layer and the j-th channel of the current layer in the image quality assessment model, respectively. Decorators are used to measure the importance of each channel, and the least important channels are pruned under the guidance of these decorators. In the split state, the input feature map F from the previous layer... i-1 With sub-convolution kernel W i * Perform as The convolution operation is then performed with the decorator α in the i-th layer. i Multiplication:
[0098]
[0099] Thus, the output feature map F i The results of the operation are the same before and after the split, and the split operation does not cause any additional errors in the reasoning results.
[0100] (2) Decorator initialization
[0101] The decorator is initialized using the Hierarchical Relevance Propagation (LRP) algorithm. The loss image x is fed into the network for inference, and activation maps of each convolutional layer are collected. The network output f(x) is propagated backward through the network according to the LRP propagation rule, and all activation maps obtain their relevance to the model output. The relevance of activations belonging to the same convolutional channel is summed to initialize the relevance of the decorator for that channel.
[0102] The specific implementation can be carried out as follows:
[0103] set up It is the correlation of neurons a at layer l. For the association of neuron b at the next layer l+1, the LRP propagation rule can be given by the following formula:
[0104]
[0105] in express arrive The correlation propagation process, z ab and z b The mapping from one layer to the next can be described by the following formula:
[0106] z ab =x a ·w ab ,
[0107]
[0108] x b =g(z) b )
[0109] Where w ab It is the weight connecting neuron a and neuron b, bias b It is the bias of layer b, z b is the pre-activation of the b-th neuron in the (l+1)-th layer, and g(·) is a non-linear activation function.
[0110] This invention selects the α-β criterion to prevent z ab When the correlation is very small, It may take unbounded values; this criterion can be described as follows:
[0111]
[0112] Where α+β=1, and “+” and “-” represent z respectively. ab and z b The positive and negative parts, in this invention, α=1, β=0;
[0113] The trained decorator can reflect the relative importance of different channels within a convolutional layer. To achieve more efficient and automated global pruning, the decorator needs to be converted into a global importance factor. This embodiment adopts a Taylor expansion-based method, expressed as:
[0114]
[0115] Where θ can be any factor related to the convolution channel. Therefore, decorator globalization can be defined as...
[0116]
[0117] Where L(·) is the loss function used to train the model, and D IQA This is the dataset to be evaluated during the pruning process, where W' represents the current model parameters, and φ ij Let be the global importance factor of the j-th channel in the i-th layer.
[0118] In another preferred embodiment, the pruned portion (corresponding to) Figure 2 The pruning and retraining module can be implemented as follows: A global importance factor is used to measure the global importance of each channel. The mask for the least important channel is set to 0, and the masks for the remaining channels are set to 1. During the pruning phase, the output of each channel is multiplied by the mask to simulate pruning.
[0119] like Figure 3 As shown, if a Batch Normalization layer follows a convolutional layer, the output of the Batch Normalization layer should be multiplied by the mask again. This setting ensures that the mask can correctly simulate the inference process of the pruned model.
[0120] Furthermore, in some embodiments, convolutional layers in the network are merged, and the operation is defined as the product between the sub-convolutional kernel and its corresponding decorator:
[0121]
[0122] Decorators will not be involved in subsequent reasoning.
[0123] The entire image quality assessment model, except for the decorator, is retrained using only the dataset to be evaluated, which mitigates the performance loss caused by pruning. After retraining, the convolutional layers are split again using the unchanged decorator; this process is iterated until the compression ratio reaches the target or the performance falls below a predefined threshold. Because a mask is used to simulate pruning during the pruning phase, an actual compact model (the pruned model) needs to be extracted based on the mask to facilitate post-pruning deployment. The extracted model is then retrained on the image quality assessment dataset; this retraining recovers the performance degradation that might have been caused by the extraction operation.
[0124] Implementation results:
[0125] To verify the effectiveness and universality of the quality assessment method of this invention, three state-of-the-art IQA models based on complex DNN structures were selected for validation: DBCNN, StairIQA with a ResNet50 backbone, and HyperIQA. All of these IQA models were independently trained and tested on three real-world distortion IQA databases.
[0126] To demonstrate the effectiveness and ease of access of the lightweight image quality assessment method of this invention, two smaller databases were selected: CLIVE and BID, and one larger database: KonIQ-10k. CLIVE consists of 1162 images taken by mobile devices with different real-world distortions. BID is a fuzzy image database containing 586 images with real-world fuzzy distortion. KonIQ-10k contains 10073 images selected from the large public multimedia database YFCC100m, covering a wide range of distortions in brightness, color, contrast, noise, sharpness, and other quality dimensions. The following two evaluation criteria were selected in the experiments to measure the performance of the objective audio-visual quality assessment method: Pearson linear correlation coefficients (PLCC) and Spearman rank order correlation coefficients (SRCC).
[0127] The image quality assessment method proposed in this invention starts with a convolutional neural network, using appropriate training strategies and pre-training parameters for each network. All experiments were implemented using the PyTorch framework on an NVIDIA RTX 3080 server.
[0128] For the database, the minimum resolution of the images is set to 380 while maintaining their aspect ratio, and images with a resolution of 320×320 are randomly cropped during the training and testing phases. The hyperparameters for the decorator training and retraining phases need to vary depending on the sparsity of the network itself.
[0129] By setting appropriate hyperparameters for decorator training and overall model retraining, and using DBCNN, StairIQA, and HyperIQA as the foundation, the application performance of the lightweight image quality assessment model is shown in Table 1.
[0130] Table 1
[0131]
[0132] On small databases: Pruning using a small IQA database is most challenging when only the IQA database is available for retraining. To verify the effectiveness of the proposed method under these conditions, three models were pruned on the small databases CLIVE and BID, and the results are shown in Table 1. The results show that as the pruning rate increases, model performance first increases and then decreases, and splitting convolutional layers during pruning can introduce performance gains to the pruned model on the small IQA database. In most cases, on small databases with only 50% of the parameters remaining, the model pruned by splitting convolutional layers can achieve the same performance as the original pruned model.
[0133] On large databases: The method of the invention was further tested on the larger KonIQ-10k database. The results show that, unlike on smaller databases, the model's performance decreases with increasing pruning rate, and no performance improvement is observed during pruning. This is reasonable because larger databases provide significantly better conditions for retraining, and the model's performance degrades much more slowly with increasing pruning rate; furthermore, after pruning 80% of the parameters, the lightweight image quality assessment model based on StairIQA and HyperIQA still performs acceptablely on the KonIQ-10k database.
[0134] In summary, experiments on several image quality assessment models demonstrate the effectiveness of the image quality assessment model pruning method of this invention. For example, when pruning 50% of the parameters, the performance of DBCNN, StairIQA, and HyperIQA can remain at the baseline level, and the models can be retrained on a real-distortion image quality assessment database containing only 586 images. Furthermore, after pruning 80% of the parameters, the performance of StairIQA and HyperIQA on the KonIQ-10k database remains acceptable.
[0135] This invention provides a lightweight image quality assessment method based on convolutional neural network pruning. By splitting the convolutional layers in the image quality assessment network into sub-convolutional kernels and decorators, it more accurately measures the importance of convolutional channels, thereby reducing the requirements for retraining after pruning. This results in an image quality assessment method with faster computation speed and smaller storage space. This method can complete image quality assessment tasks with faster computation speed and smaller storage space.
[0136] Based on the same technical concept, in another embodiment, the present invention also provides an image quality evaluation terminal based on convolutional neural network pruning, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor. When the processor executes the program, it is used to execute the image quality evaluation method based on convolutional neural network pruning.
[0137] Optionally, the memory is used to store programs; the memory may include volatile memory, such as random-access memory (RAM), such as static random-access memory (SRAM), double data rate synchronous dynamic random-access memory (DDR SDRAM), etc.; the memory may also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs, functional modules, etc. that implement the above methods), computer instructions, etc., and the aforementioned computer programs, computer instructions, etc., can be partitioned and stored in one or more memories. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by the processor.
[0138] The aforementioned computer programs, computer instructions, etc., can be stored in partitions within one or more memory locations. Furthermore, the aforementioned computer programs, computer instructions, data, etc., can be accessed by a processor.
[0139] A processor is used to execute a computer program stored in memory to implement the various steps of the methods involved in the above embodiments. For details, please refer to the relevant descriptions in the preceding method embodiments.
[0140] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.
[0141] In this embodiment of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the no-reference video evaluation method in any of the above method embodiments.
[0142] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0143] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0144] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0145] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0146] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention.
[0147] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A lightweight image quality assessment method based on convolutional neural network pruning, characterized in that, include: Obtain the dataset to be evaluated; Select a convolutional neural network structure based on the dataset to be evaluated, and train the convolutional neural network. The trained convolutional neural network is pruned to obtain an image quality assessment model; The image quality evaluation model is input into the dataset to be evaluated to obtain the image quality evaluation result; Wherein: the pruning operation on the trained convolutional neural network to obtain the image quality assessment model includes: The convolutional layers in the convolutional neural network are split into two trainable parts: sub-convolutional kernels and decorators, and the decorators are initialized using a hierarchical correlation propagation algorithm; each decorator represents the importance of the current convolutional layer. The decorator is trained using the dataset to be evaluated; Set the mask according to the decorative element and complete the pruning; The pruned convolutional neural network is fine-tuned using the dataset to be evaluated to obtain an image quality evaluation model. The convolutional layer in the convolutional neural network is split into two trainable parts: a sub-convolutional kernel and a decorator, wherein: Splitting convolutional layers is achieved through kernel factorization; The original convolution kernel for each channel Divided into sub-convolution kernels and decorative items And they are all trainable: ； in and The image quality assessment model is respectively indicated by the first... The first layer and the current layer Channels; decorators are used to measure the importance of each channel, and the least important channels are pruned under the guidance of these decorators; in the split state, the input feature map from the previous layer. sub-convolution kernel Perform as The convolution operation is then performed with the first... Decorative elements in layers Multiplication: ； Thus, the output feature map The results of the operation are the same before and after the split, and the split operation does not cause any additional errors in the reasoning results.
2. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 1, characterized in that, The step of selecting a convolutional neural network structure based on the dataset to be evaluated and training the convolutional neural network includes: Based on the dataset to be evaluated, a convolutional neural network structure is selected, and image classification is used as the source task to complete pre-training on a public dataset. The pre-trained convolutional neural network is further trained on the dataset to be evaluated to complete transfer learning; wherein, the feature extraction part of the pre-trained convolutional neural network is connected to a new regressor to perform transfer learning on image quality evaluation, and the semantic knowledge in the feature extractor is used to improve the performance of image quality evaluation.
3. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 1, characterized in that, The step of training the decorator using the dataset to be evaluated includes: During the pruning phase, each part of the convolutional neural network, except for the decorator, is frozen; The decorator is trained using the dataset to be evaluated to better adapt it to the image quality evaluation task; The trained decorator is converted into a global importance factor for each channel using Taylor expansion, and a pruning mask is set under the guidance of the importance factor.
4. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 3, characterized in that, The step of setting a mask based on the decorator and completing pruning includes: The importance of each channel is measured using the global importance factor, the mask of the least important channel is set to 0, and the masks of the remaining channels are set to 1; during the pruning phase, the output of each channel is multiplied by the mask to simulate pruning.
5. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 4, characterized in that, The step of fine-tuning the pruned convolutional neural network using an image quality assessment dataset to obtain an image quality assessment model includes: The convolutional layers in the convolutional neural network are merged, and the entire convolutional neural network except for the decorator is retrained using the dataset to be evaluated to mitigate the performance loss caused by pruning; after retraining, the convolutional layers are split again using the unchanged decorator. The pruning operation is iterated until the compression ratio reaches the target or the performance is lower than a predefined threshold, thus obtaining the image quality evaluation model.
6. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 5, characterized in that, The merging of convolutional layers in the convolutional neural network is to restore the convolutional layers to their normal state, wherein the "merge" operation is defined as the product between the sub-convolutional kernel and its corresponding decorator.
7. The lightweight image quality assessment method based on convolutional neural network pruning according to claim 1, characterized in that, The decorator is initialized using a hierarchical dependency propagation algorithm, wherein: Lost image The data is fed into the convolutional neural network for inference and the activation map of each convolutional layer is collected. The output of the convolutional neural network The results, based on the LRP propagation rule, propagate backward through the network, and all activation maps obtain their correlation with the output of the image quality assessment model; The correlations of activations belonging to the same convolutional channel are summed to initialize the correlation of the decorator for that channel.
8. A lightweight image quality assessment system based on convolutional neural network pruning, characterized in that, include: Image acquisition module: Acquires the dataset to be evaluated; Model pre-training module: Selects a convolutional neural network structure based on the dataset to be evaluated, and trains the convolutional neural network; Model pruning module: Prunes the trained convolutional neural network to obtain an image quality assessment model; Evaluation module: Input the dataset to be evaluated into the image quality evaluation model to obtain the image quality evaluation result; Wherein: the model pruning module includes: The convolutional layer in the convolutional neural network is split into two trainable parts: a sub-convolutional kernel and a decorator, and the decorator is initialized using the hierarchical correlation propagation algorithm; The decorator is trained using the dataset to be evaluated; Set the mask according to the decorative element and complete the pruning; The pruned convolutional neural network is fine-tuned using the dataset to be evaluated to obtain an image quality evaluation model. The model pruning module splits the convolutional layers in the convolutional neural network into two trainable parts: sub-convolutional kernels and decorators, wherein: Splitting convolutional layers is achieved through kernel factorization; The original convolution kernel for each channel Divided into sub-convolution kernels and decorative items And they are all trainable: ； in and The image quality assessment model is respectively indicated by the first... The first layer and the current layer Channels; decorators are used to measure the importance of each channel, and the least important channels are pruned under the guidance of these decorators; in the split state, the input feature map from the previous layer. sub-convolution kernel Perform as The convolution operation is then performed with the first... Decorative elements in layers Multiplication: ； Thus, the output feature map The results of the operation are the same before and after the split, and the split operation does not cause any additional errors in the reasoning results.
9. A lightweight image quality evaluation terminal, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the program, it is used to perform the method described in any one of claims 1-7.