A compressed image recognition method based on deep learning

Abstract

The present invention discloses a compressed image recognition method based on deep learning, wherein a trained feature enhancement module is embedded into a pre-trained classification model to obtain a deep neural network model for compressed image recognition; then, the compressed image to be recognized is input into the deep neural network model for classification; finally, the final classification result is output. The feature enhancement module embedded in the present invention does not affect any parameters of the original pre-trained model, and the feature enhancement module added in the middle can effectively improve the classification performance of the compressed image without consuming a lot of time for retraining.

Description

Technical Field

[0001] The present invention relates to the technical field of image compression, and in particular to a compressed image recognition method based on deep learning. Background Art

[0002] JPEG (Joint Photographic Experts Group) compression technology is widely used in mobile devices. Its design purpose is to achieve a compromise between bit rate cost savings and visual quality by sacrificing imperceptible high-frequency components, reducing the bandwidth of image transmission and the size of local file storage without affecting the human eye's perception as much as possible. With the development of artificial intelligence technology, captured images are not only used for human perception, but also for visual analysis such as image classification. Among them, image classification is to distinguish different categories of images based on the semantic information of the image. It is a basic problem in computer vision and the basis of high-level visual tasks such as image detection, image segmentation, object tracking, etc.

[0003] Existing pre-trained models are usually trained on a large, fresh image dataset of high quality. Therefore, embedded visual analysis models in mobile devices are usually pre-trained and only apply to actual images with distributions close to those in the training set. However, the images used in real scenes come from a variety of different compression ratios. When the compression ratio of the input image exceeds the range of the training set, the performance of the pre-trained model will be greatly reduced. The main reason for the decline in compressed image recognition performance is the uneven feature drift caused by block compression during the compression process. JPEG compression technology divides the image into 8×8 blocks and applies discrete cosine transform (DCT) and quantization to each block. However, due to the different structures and statistical characteristics of different image blocks, the high-frequency components lost in each block are also different, which will lead to inconsistent feature drift between blocks, and then produce spatially different feature drift on the compressed image, which will eventually affect subsequent image recognition tasks and cause the problem of reduced recognition accuracy.

[0004] At present, the typical method to solve the recognition accuracy of compressed images is a two-step method, that is, first use image compression artifact removal technology to reduce artifacts such as block effects generated during the compression process, and then perform subsequent image recognition on the enhanced image. However, common artifact removal techniques are not specifically designed to restore imperceptible tiny textures, which will affect the statistical consistency of the image and limit the improvement of recognition performance. In addition, the smoothing operation in artifact removal will lead to the loss of details in low compression ratio images, which will aggravate the feature drift in the compressed image and reduce the recognition performance. Summary of the invention

[0005] In the prior art, compressed images will produce obvious block effects in the pixel space, resulting in bias in subsequent feature extraction, and the offset features will affect the final image recognition performance. Considering that only enhancing compressed images from the pixel space will only bring limited performance improvement to subsequent image recognition tasks. The present invention provides a compressed image recognition method based on deep learning, which can be plug-and-play embedded in a pre-trained classification model, effectively improving the recognition performance of the classification model on compressed images.

[0006] In a first aspect, the present invention protects a method for constructing and training a feature enhancement module, comprising the following steps:

[0007] Step 1: Select the pre-trained classification model to be enhanced and the corresponding feature position.

[0008] Furthermore, the pre-trained classification model can use commonly used classic architectures, such as VGG16 network and SqueezeNet network. When using the VGG16 network, the Conv2-2 layer of the network is selected as the location for feature enhancement; when using the SqueezeNet network, the first layer of the network is selected as the location for feature enhancement.

[0009] Step 2, construct a feature enhancement network (FE-Net);

[0010] Furthermore, the feature enhancement network can adopt a network that integrates multi-level features, with a total of 7 layers; among which, the first 6 layers adopt 3*3 convolution and the last layer adopts 1*1 convolution.

[0011] Step 3, input the compressed image, and output the feature deviation map according to the adaptive feature deviation estimation network (FDE-Net);

[0012] Furthermore, the step 3 is specifically as follows: first, the compressed image is subjected to DCT transformation; then, the DCT coefficients within the block are modeled through a convolutional neural network in the converted DCT domain; finally, the mapping relationship between the DCT coefficient distribution and the feature deviation degree is learned to obtain a feature deviation map.

[0013] Step 4: Input the feature deviation map into the feature enhancement network, output the enhanced features, and the feature enhancement module is constructed.

[0014] Furthermore, the step 4 is specifically as follows: use 1*1 convolution to model the DCT coefficients, obtain three feature deviation maps in three levels, and apply them to the front, middle and back positions of the feature enhancement network respectively for multi-level enhancement.

[0015] The training of the feature enhancement module is specifically as follows: inputting the compressed training image into the feature enhancement module constructed above, outputting the enhanced features of the compressed training image, and using the features of the original training image to supervise the enhanced features;

[0016] The compressed training images and original training images are sample sets collected in advance for training.

[0017] Furthermore, the loss function in the supervised learning adopts the mean square error loss function, the training adopts the Adam optimizer, the batch-size is 32, and the initial learning rate is 0.0001.

[0018] Furthermore, the compressed training image is obtained by compressing the original training image using different compression factors (QF) to obtain compressed training images with different compression qualities of high, medium and low; wherein QF is selected as 65, 50, 25, 18, 15, 10, 7, a total of 6 levels of compression quality.

[0019] In the second aspect, the present invention protects a compressed image recognition method based on deep learning. First, the feature enhancement module trained in the first aspect is embedded into a pre-trained classification model to obtain a deep neural network model for compressed image recognition; then, the compressed image to be recognized is input into the deep neural network model for classification; finally, the final classification result is output.

[0020] The feature enhancement module is used to correct the features of the compressed image offset in the shallow feature space to achieve enhancement in the compressed image feature space.

[0021] The beneficial effects of the present invention are as follows: 1. The feature enhancement module embedded in the present invention does not affect any parameters of the original pre-trained model, and the feature enhancement module added in the middle can effectively improve the classification performance of the compressed image without spending a lot of time on retraining; 2. The present invention enhances the compressed image in the feature space, and designs a feature deviation estimation network to estimate the feature deviation degree of the compressed image in the DCT domain, thereby more accurately enhancing the compressed image features; 3. The present invention is trained on data sets with multiple compression degrees, and can be plug-and-play embedded in various pre-trained classification models, thereby effectively improving the recognition accuracy of the pre-trained model for the compressed image. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of a compressed image recognition method based on deep learning;

[0023] Figure 2 This is a schematic diagram of the construction process of the feature enhancement module in Example 1;

[0024] Figure 3This is a schematic diagram of training data synthesis in Example 1;

[0025] Figure 4 Schematic diagram of the training process in Example 1;

[0026] In the figure: (a) is the original lossless image, (b) is the compressed image, (c) is the damaged feature map, (d) is the enhanced residual result map, (e) is the feature map after image enhancement, and (f) is the feature map of the original lossless image.

[0027] Figure 5 Schematic diagram of some enhanced features in Example 1.

[0028] In the figure: (a) is the original lossless image, (b) is the compressed image, (c) is the clear feature map, (d) is the damaged feature map of the compressed image, and (e) is the feature map after image enhancement. DETAILED DESCRIPTION

[0029] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention are provided for the purpose of illustration and description, and are not intended to be exhaustive or to limit the present invention to the disclosed forms. Many modifications and variations will be apparent to those of ordinary skill in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thereby design various embodiments with various modifications suitable for specific uses.

[0030] Example 1

[0031] This embodiment combines the lossless image dataset DIV2K containing a large number of natural scenes. The data is saved in PNG format, and a total of 2,000 uncompressed lossless original images are used as original training images. Figure 1 ) are described in detail.

[0032] First, design feature enhancement modules, such as Figure 2 As shown, the following steps are included:

[0033] Step 1: Select the pre-trained classification model to be enhanced and the corresponding feature position.

[0034] Specifically, the pre-trained classification model can use commonly used classic architectures, such as VGG16 network and SqueezeNet network. When using the VGG16 network, the Conv2-2 layer of the network is selected as the location for feature enhancement; when using the SqueezeNet network, the first layer of the network is selected as the location for feature enhancement.

[0035] The VGG16 network has 16 layers in total and is a fairly deep convolutional neural network. VGG structures at all levels use 5 convolutions, each with one or more convolution layers. At the same time, each segment is followed by a maximum pooling layer to reduce the image size. The number of convolution kernels in each segment is the same, and the more convolution kernels there are at the end. The convolution series in the VGG16 network has fewer parameters than using a single larger convolution kernel, and has more nonlinear transformations than a single convolution layer.

[0036] The SqueezeNet network is a lightweight and efficient CNN model with 50x fewer parameters than AlexNet, but the model performance is close to AlexNet. Under acceptable performance, small models have many advantages over large models: (1) more efficient distributed training, small models have small parameters, and reduced network communication; (2) easy model update, small models, and easy client program update.

[0037] Step 2, construct a feature enhancement network (FE-Net);

[0038] Specifically, the feature enhancement network can adopt a network that integrates multi-level features, with a total of 7 layers; among them, the first 6 layers use 3*3 convolution to extract features of different receptive fields, and the output dimensions are 128, 128, 64, 64, 32, and 32 respectively; the last layer uses 1*1 convolution to ensure that the output dimension remains unchanged from the feature dimension of the initial input, and the output dimension is 128 dimensions.

[0039] Step 3: Input the compressed image and output the feature deviation map according to the adaptive feature deviation estimation network (FDE-Net). Since the compressed image will have different degrees of bias in different areas, it is necessary to estimate a feature deviation map to assist FE-Net in targeted enhancement of global features.

[0040] Specifically, the step 3 is as follows: first, the compressed image is subjected to DCT transformation; then, the DCT coefficients in the block are modeled through a convolutional neural network in the converted DCT domain; finally, the mapping relationship between the DCT coefficient distribution and the feature deviation degree is learned to obtain a feature deviation map.

[0041] Step 4: Input the feature deviation map into the feature enhancement network, output the enhanced features, and the feature enhancement module is constructed.

[0042] Specifically, step 4 is as follows: modeling the DCT coefficients using 1*1 convolution, obtaining three feature deviation maps in three levels, and applying them to the front, middle and back positions of the feature enhancement network for multi-level enhancement. By multiplying the feature deviation map with the feature enhancement network, the features of different regions are enhanced in a targeted manner, thereby achieving accurate enhancement of the compressed image features.

[0043] Then, the constructed feature enhancement module is trained, specifically: the compressed training image is input into the constructed feature enhancement module, the enhanced features of the compressed training image are output, and the enhanced features are supervised and learned using the features of the original training image;

[0044] Among them, 2000 lossless original images are compressed using different compression factors QF (Quality Factor) to obtain compressed images with different compression qualities of high, medium and low. Here, QF is selected as 65, 50, 25, 18, 15, 10, and 7, a total of 6 compression qualities, see Figure 3 , a total of 12,000 256*256 lossless-compressed image training pairs are generated for training the feature enhancement module.

[0045] Take one of the original training images as an example to show the training process. Figure 4 As shown in the figure, the performance of the feature module in subsequent tasks is improved through the training process.

[0046] Specifically, the loss function in the supervised learning adopts the mean square error loss function, the training adopts the Adam optimizer, the batch-size is 32, and the initial learning rate is 0.0001.

[0047] Finally, the trained feature enhancement module is embedded into the pre-trained classification model to obtain a deep neural network model for compressed image recognition; the compressed image to be recognized is input into the deep neural network model for classification, and the final classification result is output.

[0048] like Figure 5 As shown, three pictures in the original training images are used as the test set. It can be seen that after being enhanced by the method of the present invention, the recognizability of the features is greatly improved.

[0049] Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field and related fields without creative work should fall within the scope of protection of the present invention.

Claims

1. A compressed image recognition method based on deep learning, characterized in that: The steps include: Step 1: Select the pre-trained classification model to be enhanced and the corresponding feature position; Step 2, construct the feature enhancement network FE-Net; Step 3, input the compressed image, and perform DCT transformation on the compressed image according to the adaptive feature deviation estimation network FDE-Net; then, the DCT coefficients in the block are modeled by the convolutional neural network in the transformed DCT domain; finally, the mapping relationship between the distribution of DCT coefficients and the degree of feature deviation is learned to obtain the feature deviation map; Step 4: Input the feature deviation map into the feature enhancement network, output the enhanced features, and the feature enhancement module is constructed. Specifically, the DCT coefficients are modeled using 1*1 convolution, and three feature deviation maps are obtained in three levels. They are applied to the front, middle and back positions of the feature enhancement network for multi-level enhancement. Step 5: train the feature enhancement module and embed the trained feature enhancement module into the pre-trained classification model to obtain a deep neural network model for compressed image recognition; then, input the compressed image to be recognized into the deep neural network model for classification; finally, output the final classification result.

2. The compressed image recognition method based on deep learning according to claim 1, characterized in that: The pre-trained classification model is any one of the VGG16 network and the SqueezeNet network; Among them, when using the VGG16 network, the Conv2-2 layer of the network is selected as the location of feature enhancement; when using the SqueezeNet network, the first layer of the network is selected as the location of feature enhancement.

3. The compressed image recognition method based on deep learning according to claim 1, characterized in that: The feature enhancement network may adopt a network that fuses multi-level features, with a total of 7 layers; Among them, the first 6 layers use 3*3 convolution, and the last layer uses 1*1 convolution.

4. The compressed image recognition method based on deep learning according to any one of claims 1 to 3, characterized in that: The training of the feature enhancement module is specifically as follows: inputting the compressed training image into the constructed feature enhancement module, outputting the enhanced features of the compressed training image, and using the features of the original training image to perform supervised learning on the enhanced features; The compressed training images and original training images are sample sets collected in advance for training.

5. The compressed image recognition method based on deep learning according to claim 4 is characterized in that: The loss function in the supervised learning adopts the mean square error loss function, the training adopts the Adam optimizer, the batch-size is 32, and the initial learning rate is 0.0001.

6. The compressed image recognition method based on deep learning according to claim 4, characterized in that: The compressed training images are obtained by compressing the original training images using different compression coefficients to obtain compressed training images with different compression qualities of high, medium and low. Among them, QF is selected as 65, 50, 25, 18, 15, 10, and 7, a total of 6 levels of compression quality.

Images