A 2s-fold super-resolution restoration device for JPG image data entities
Through lightweight CNN and multi-stage processing technology, the noise problem of JPG image data in the super-resolution model is solved, the resolution and perception effect of the image is improved, and it is suitable for a variety of image scenes, reducing equipment costs, and solving the current situation There are problems in the technology that image processing costs and poor results are high.
Patent Information
- Application Number
- CN202210352479.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-04-05
- Publication Date
- 2024-11-05
- Estimated Expiration
- 2042-04-05
AI Technical Summary
The existing super-resolution model is not effective when processing JPG compressed image data, especially inability to effectively remove checkerboard noise, resulting in excessive smoothing of image details and unable to meet actual needs. The traditional model is expensive and cannot be applied to multiple image scenarios.
Lightweight convolutional neural network (CNN) is used to train the compressed information of JPG image data, and it is processed through multi-stage processing, including image noise removal, high accuracy recovery and high perception recovery, using specific loss functions and model structures, combined with L1 Norm, perceptual loss and GAN generation confrontation loss gradually improve image resolution.
Effectively remove noise in JPG images, improve the peak signal-to-noise ratio (PSNR) and perceptual effect of the image, generate high-resolution images that meet actual needs, reduce equipment costs, and are suitable for a variety of image scenarios, especially in the knowledge graph. Image data processing.
Smart Images

Figure CN114708146B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of super-resolution restoration of data, and in particular to a super-resolution restoration device for a JPG type image data entity, and in particular to a 2S times super-resolution restoration device for a JPG image data entity. Background Art
[0002] Artificial intelligence has become the hottest topic in current social development. At present, the development of intelligent technology is mainly focused on image tasks and natural language processing related to unstructured data. Knowledge graphs can provide a unified platform for a variety of different data, and further provide a data analysis basis for intelligent tasks involving a variety of different data types.
[0003] Image data is important data involved in knowledge graphs. Due to its unstructured data characteristics, how to process image data is one of the important tasks of knowledge graphs.
[0004] For images, pixels are the basic elements of various tasks and requirements, and limited pixel values will constrain the further processing space of the image. The purpose of the present invention is to solve the problem of obtaining high-pixel images under limited conditions, and further serve other decision-making tasks or application purposes. Based on the two different purposes of high accuracy and high perception, different neural network models are set. Through the mutual combination and matching of models, the most suitable high-pixel value images can be matched for different purposes. Through the model of the present invention, a better balance can be achieved between the requirements and the limited hardware conditions, and the information content and application scope of the image data under limited conditions can be further improved.
[0005] Image and speech recognition currently mainly use deep model learning to fit the distribution of unstructured image, speech and text data. Limited by the objective environment, the quality of data varies greatly. How to better process this data information has a significant impact on the results of the model. This patent mainly focuses on how to improve the pixel information of image data and construct a high-quality target image.
[0006] As one of the most popular primary tasks in computer vision, super-resolution has always been a hot topic in academia and industry. Current research focuses on two aspects: how to improve the accuracy of generated images (evaluation indicators Peak Signal-to-Noise Ratio (PSNR) and Structural Similarity (SSIM)) and how to improve the perceptual effect of images (image similarity metric LPIPS). Most pixel generation models are related to deep learning, which has made rapid progress compared to traditional methods.
[0007] At present, most of the research models are based on standard data sets in ideal conditions, which are very different from the various images in reality. In addition, for the widely existing JPG compressed image data, the performance of traditional super-resolution models cannot achieve satisfactory results. Summary of the invention
[0008] In order to solve the above problems, the technical solution adopted by the present invention is:
[0009] A 2s-fold super-resolution restoration device for a JPG image data entity, characterized in that it includes a computer program having the following functions:
[0010] Image noise processing;
[0011] High-precision recovery;
[0012] Image high-perceptual restoration.
[0013] Preferably, a lightweight convolutional neural network (CNN) is used to train and restore the compressed information of the JPG image data entity.
[0014] Preferably, the compression information of the JPG image data entity is an input image obtained by compressing a lossless 1X image through JPG;
[0015] The input image is passed through a multi-layer convolutional neural network, and the information is fully learned and extracted, resulting in a denoised image (1X) with compressed information partially restored;
[0016] Among them, the loss function is set to the L1 norm of the JPG compressed image and the lossless 1X image:
[0017]
[0018] Preferably, the high-precision recovery process includes:
[0019] The input compressed JPG image undergoes the first stage of noise processing to obtain an original size image that meets the usage requirements. The output data is used as the input data of the second stage for the next super-resolution operation.
[0020] Preferably, the super-resolution operation process includes:
[0021] a, the output of the first stage Lrec(I C-JPG ) is used as the original input image. The three channels of RGB are passed through the convolutional neural network. Through a large number of convolution operations, activation functions (Leaky-ReLU), local residual modules and other operations, the information of the input data is fully extracted and transmitted, and the low-dimensional information is gradually transformed into high-dimensional information.
[0022] b, the n-layer information obtained at the end of stage a, through a layer of convolution operation, obtains the n-fold magnification of the target s s Based on the multi-layer information, the target image I corresponding to s times the size is obtained through pixel difference operation. s1 ;
[0023] c, obtained I s1 And supervised image downsampling (Bicubic) to the image I of the corresponding size in this stage HR’ The training data pair is formed. The final loss function of this stage is set as:
[0024] L scale1 =|I s1’ I HR’ |;
[0025] After the second stage of training, the denoised original image has completed the initial s-fold super-resolution amplification. The final image data can be output as intermediate data to meet the super-resolution task with high accuracy requirements. The evaluation indicator for this part is only the peak signal-to-noise ratio (PSNR).
[0026] The beneficial effect is that: focusing on how to process JPG image data in reality, by combining specific needs, setting a super-resolution model with satisfactory effect. By iterating and combining high-precision and high-perception super-resolution models, the noise interference in JPG images is eliminated, and super-resolution images that meet actual needs are further generated. The device can only obtain high-resolution images for new image data. For existing, especially long-ago photos, new devices cannot process them. In addition, the price of high-definition image equipment is often extremely expensive, and for many users, its use cost is too high. This method can solve this problem well from the perspective of the model. The traditional super-resolution model processes images based on a pre-trained model. The existing super-resolution models are mainly divided into high precision (measured by peak signal-to-noise ratio PSNR and similarity SSIM) and high perception (currently there is no standard measurement indicator). At present, there is very little research on setting corresponding super-resolution models for different scenes. In view of this situation, the present invention uses professional data and purpose of use analysis, uses different weight ratios, and resets a new super-resolution model. The use of the model is relatively simple. The present invention is an end-to-end model based on deep learning. It only needs to input an image to obtain a corresponding high-resolution image, which is convenient and practical. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 Shown is an example of a JPG image used in an embodiment of the present invention;
[0028] Figure 2The figure is a flowchart of restoring a super-resolution image by a 2s-fold super-resolution restoration device of a JPG image data entity according to an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The following embodiments are proposed based on the inventive concept of the present invention and are aimed at a specific problem scenario, and should not be understood as limiting the protection scope of the present invention.
[0030] Embodiment 1:
[0031] A 2s-fold super-resolution restoration device for a JPG image data entity, characterized in that it includes a computer program having the following functions:
[0032] Image noise processing;
[0033] High-precision recovery;
[0034] Image high-perceptual restoration.
[0035] Embodiment 2:
[0036] A lightweight convolutional neural network (CNN) is used to train and restore the compressed information of JPG image data entities.
[0037] Embodiment 3:
[0038] The compression information of the JPG image data entity is an input image obtained by compressing a lossless 1X image through JPG;
[0039] The input image is passed through a multi-layer convolutional neural network, and the information is fully learned and extracted, resulting in a denoised image (1X) with compressed information partially restored;
[0040] Among them, the loss function is set to the L1 norm of the JPG compressed image and the lossless 1X image:
[0041]
[0042] Embodiment 4:
[0043] The high-precision recovery process includes:
[0044] The input compressed JPG image undergoes the first stage of noise processing to obtain an original size image that meets the usage requirements. The output data is used as the input data of the second stage for the next super-resolution operation.
[0045] Embodiment 5:
[0046] The super-resolution operation process includes:
[0047] a, the output of the first stage Lrec(I C-JPG) is used as the original input image. The three channels of RGB are passed through the convolutional neural network. Through a large number of convolution operations, activation functions (Leaky-ReLU), local residual modules and other operations, the information of the input data is fully extracted and transmitted, and the low-dimensional information is gradually transformed into high-dimensional information.
[0048] b, the n-layer information obtained at the end of stage a, through a layer of convolution operation, obtains the n-fold magnification of the target s s Based on the multi-layer information, the target image I corresponding to s times the size is obtained through pixel difference operation. s1 ;
[0049] c, obtained I s1 And supervised image downsampling (Bicubic) to the image I of the corresponding size in this stage HR’ The training data pair is formed. The final loss function of this stage is set as:
[0050] L scale1 =|I s1’ I HR’ |;
[0051] After the second stage of training, the denoised original image has completed the initial s-fold super-resolution amplification. The final image data can be output as intermediate data to meet the super-resolution task with high accuracy requirements. The evaluation indicator for this part is only the peak signal-to-noise ratio (PSNR).
[0052] In the Internet age, images are mainly stored in JPG format. This format can retain people's visual experience to the greatest extent, while greatly compressing the storage space of images. In the Internet age, massive image data occupies a lot of storage space and transmission bandwidth. Therefore, JPG, which can compress image size while retaining most visual experience, has become the mainstream. At the same time, most of the current mainstream super-resolution models are pre-trained based on ideal training data sets, and the processing of some real photo data also considers the differences brought by photosensitive elements, which is not suitable for JPG image super-resolution tasks. This is because the super-resolution model cannot recognize the checkerboard noise generated by the JPG compression process, and defaults to image information, which will amplify the visual noise during the super-resolution process.
[0053] This section intends to use a lightweight convolutional neural network (CNN) to train the recovery of JPG compressed information. The process is shown in stage 1 in the figure. Specifically, the input JPG image is obtained by JPG compression (with multiple different compression ratios) based on the lossless 1x (1X) image; the input image passes through a multi-layer convolutional neural network, and the information is fully learned and extracted to obtain a noisy image (1X) with partially restored compressed information. At this stage, the loss function is set to the L1 norm of the JPG compressed image and the lossless 1X image:
[0054]
[0055] The final model can be described as: ILR = NN (IC-JPG).
[0056] After learning in the first stage, the JPG image with checkerboard noise can remove noise information to the greatest extent, and the lost detail information can be partially restored, meeting the next step requirements of real image super-resolution.
[0057] High-precision recovery phase
[0058] The input compressed JPG image is processed by the first stage of noise area, and the original size image that meets the use requirements is obtained. The output data is used as the input data of the second stage for the next super-resolution operation. In this section, taking the 4x enlargement size as an example, the second stage only corresponds to half of the 2x enlargement requirement.
[0059] Generally speaking, traditional super-resolution models use L1 norm (absolute value) or L2 norm (square root of elements) as loss functions to measure the differences in the corresponding pixel values of the image. This is what is usually called a high-precision model. Later, with the rise of deep learning, perceptual loss was introduced into the super-resolution field, and a high-perceptual super-resolution model was further extended. This part will be elaborated in detail in the third stage. In this part, we focus on how to restore high-precision super-resolution images. The specific design is as follows:
[0060] a, the output of the first stage Lrec(I C-JPG ) is used as the original input image. The three channels of RGB pass through the convolutional neural network. Through a large number of convolution operations, activation functions (Leaky-ReLU), local residual modules and other operations, the information of the input data is fully extracted and transmitted, and the low-dimensional information is gradually transformed into high-dimensional information.
[0061] b. The n layers of information obtained at the end of stage a are converted into n layers of target magnification s through a convolution operation. s Based on the multi-layer information, the target image I corresponding to s times the size is obtained through pixel difference operation. s1 .
[0062] c, obtained I s1 And supervised image downsampling (Bicubic) to the image I of the corresponding size in this stage HR' The training data pair is formed. The final loss function of this stage is set as:
[0063] L scalel =|I s1’ I HR' |.
[0064] After the second stage of training, the denoised original image has completed the initial s-fold super-resolution amplification. The final image data can be output as intermediate data to meet the super-resolution task with high accuracy requirements. The evaluation indicator for this part is only the peak signal-to-noise ratio (PSNR).
[0065] Image high-perception restoration
[0066] After the second stage of processing, the image has been initially enlarged by s times. The pixel value becomes 2s times of the input data. Through the denoising model in the first stage and the high-precision (high PSNR) model in the second stage, the image data obtained is clean and the pixel values are relatively consistent. However, one problem with such an image is that the details are overly smoothed, which gives people an unrealistic visual experience as a whole. In order to further increase the detail information of the image, the third stage introduces perceptual loss, as follows:
[0067] After the second stage, the s-fold amplification of I s1 As the input data of the third stage, as in the second stage, after a series of operations from the three channels of RGB, the information of the input data is fully extracted and transmitted, and gradually transformed from low-dimensional information to high-dimensional information. It should be pointed out here that as the image size increases, the demand for model parameters and the amount of computation consumed will increase. Therefore, the specific structure of the model will be different from the models in the first two stages.
[0068] As in the second stage, the information obtained is transformed and PixelShuffled to obtain the final second s-fold magnified image Is2.
[0069] Unlike the second stage, the loss function of this stage includes three parts: L1 norm loss, perceptual loss, and GAN generation adversarial loss. The specific setting is to use the model of the third stage as the GAN generation network and jointly train it with the set discriminant network to form the basic framework of the model. The training of the entire GAN model is based on the loss function of the standard GAN.
[0070] At the same time, the obtained 2s times magnified Is2 and the original image of the supervisory information (HR) are used as input information. The representation information of the selected middle layers is extracted through the pre-trained VGG19 model, and the perceptual loss is obtained by calculating the Euclidean distance of the representation information;
[0071] The absolute value difference between the final image and the original HR image must be calculated, that is, the L1 loss. The final model loss at this stage includes: Loss-GAN, Loss-VGG, Loss-L1. We set specific hyperparameters for each part to balance the impact of different losses. The final loss function is:
[0072] Loss=a*Loss-GAN+b*Loss-VGG+c*Loss-L1;
[0073] After three stages, the final model can obtain a super-resolution image ((H*2s)*(W*2s)) from the original input image (H*W), and the overall image pixel value increases by (4s 2 ) times. At the same time, the weight of the loss function can be adjusted according to different purposes to obtain s-fold images or 2s high-precision images. This can play a good role in compensating for the lack of image data in the knowledge graph.
[0074] In the description of the embodiments of the present invention, it should be understood that the terms "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "center", "top", "bottom", "top", "bottom", "inside", "outside", "inner side", "outer side" and the like indicate orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, and are only for the purpose of facilitating the description of the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Among them, "inside" refers to an internal or enclosed area or space. "Periphery" refers to the area surrounding a specific component or a specific area.
[0075] In the description of the embodiments of the present invention, the terms "first", "second", "third", and "fourth" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", "third", and "fourth" may explicitly or implicitly include one or more of the features. In the description of the present invention, unless otherwise specified, "multiple" means two or more.
[0076] In the description of the embodiments of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "install", "connect", "connect", and "assemble" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a direct connection, or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0077] In the description of the embodiments of the present invention, specific features, structures, materials or characteristics may be combined in a suitable manner in any one or more embodiments or examples.
[0078] In the description of the embodiments of the present invention, it is to be understood that "-" and "~" represent the range between two values, and the range includes the endpoints. For example: "AB" represents a range greater than or equal to A and less than or equal to B. "A~B" represents a range greater than or equal to A and less than or equal to B.
[0079] In the description of the embodiments of the present invention, the term "and / or" herein is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " herein generally indicates that the associated objects before and after are in an "or" relationship.
[0080] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.
Claims
1. A 2s-fold super-resolution restoration device for JPG image data entities, characterized in that: include, A computer program that performs the following functions: Image noise processing; wherein, a lightweight convolutional neural network is used to train and restore the compression information of the JPG image data entity; the compression information of the JPG image data entity is an input image obtained by compressing a lossless 1x image through JPG; The input image is passed through a multi-layer convolutional neural network, and the information is fully learned and extracted, resulting in a denoised image with some compressed information restored. Among them, the loss function is set to the L1 norm of the JPG compressed image and the lossless 1X image: High-precision restoration: The high-precision restoration process includes: the input compressed JPG image undergoes the first stage of noise processing to obtain the original size image that meets the use requirements, and the output data is used as the input data of the second stage for the next super-resolution operation; the super-resolution operation process includes: a, The output Lrec of the first stage is used as the original input image. It passes through the convolutional neural network from the three channels of RGB. Through a large number of convolution operations, activation functions, and local residual module operations, the information of the input data is fully extracted and transmitted, and the low-dimensional information is gradually transformed into high-dimensional information; b, the n-layer information obtained at the end of stage a, through a layer of convolution operation, obtains the n-fold magnification of the target s s Layer information; Based on the multi-layer information, the target image I corresponding to s times the size is obtained through pixel difference operation s1 ; c, obtained I s1 And the supervised image is downsampled to the image I of the corresponding size in this stage HR’ The training data pairs are formed; the final loss function of this stage is set as: L scale1 =|I s1 ,I HR’ | After the second stage of training, the denoised original image has completed the initial s-fold super-resolution enlargement; the final image data can be output as intermediate data to meet the super-resolution task with high accuracy requirements. The evaluation indicator of this part is only the peak signal-to-noise ratio; High-perceptual image restoration; the loss function of this stage includes three parts: L1 norm loss, perceptual perception loss, and GAN generation adversarial loss; the model of the third stage is set as the generation network of GAN, and is jointly trained with the set discriminant network to form the basic framework of the model. The training of the entire GAN model is based on the loss function of the standard GAN; The obtained 2s times magnified Is2 and the original image of the supervision information are used as input information. The selected representation information in the middle is extracted through the pre-trained VGG19 model, and the perceptual loss is obtained by calculating the Euclidean distance of the representation information. The absolute value difference between the final image and the original HR image is calculated, and the L1 loss is used. The final model loss at this stage includes: Loss-GAN, Loss-VGG, and Loss-L1. Specific hyperparameters are set for each part to balance the impact of different losses. The final loss function is: Loss=a*Loss-GAN+b*Loss-VGG+c*Loss-L1.
Citation Information
Patent Citations
Image super-resolution reconstruction method and system based on edge detection
CN111062872A
Super-resolution reconstruction method
CN111899161A
Super resolution using a generative adversarial network
US20180075581A1
