A super-resolution method, medium and device based on machine learning

Abstract

The present invention provides a super-resolution method, medium and system based on machine learning, wherein the super-resolution method based on machine learning comprises: obtaining content statistical characteristics of video images; classifying the video images according to the content statistical characteristics of the video images to obtain at least one group; selecting a corresponding machine learning model for super-resolution reconstruction according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video images to obtain a corresponding high-resolution image. According to the super-resolution method based on machine learning of the present invention, compression noise removal and image super-resolution reconstruction can be achieved in one step, thereby improving the image quality obtained by super-resolution.

Description

Technical Field

[0001] The present invention belongs to the field of image enhancement and relates to a super-resolution method, and in particular to a super-resolution method, medium and system based on machine learning. Background Art

[0002] Due to the limitation of transmission bandwidth, common Internet videos generally have a smaller resolution and a lower bit rate. When these Internet videos are displayed on a large screen, ordinary differential amplification of small-resolution images will lead to problems such as edge jaggedness and blurred details. Super-resolution is usually required to reduce the high-frequency loss caused by amplification. Image super-resolution (SR) refers to the restoration of high-resolution images (HR) from low-resolution images (LR) or image sequences. HR means that the image has a high pixel density and can provide more details, which often play a key role in applications. The process of obtaining a high-resolution image through a series of low-resolution images is super-resolution reconstruction. Traditional super-resolution algorithms usually include difference-based and learning-based methods, such as SRCNN (Super-Resolution Convolutional Neural Network), DRRN (Deep Recursive Residual Network), and SRGAN (Single Image Super-Resolution Using a Generative Adversarial Network).

[0003] However, low-resolution compressed videos generally have image quality problems such as blurred details, block effects, ringing noise, and mosquito noise. If super-resolution and detail enhancement are performed directly on the image, while supplementing the high-frequency details lost due to amplification, the noise may also be supplemented as details, making it impossible to ideally filter out the subsequent denoising module. If denoising is performed first, some details are filtered out as noise, making it difficult to restore more details with subsequent super-resolution. Therefore, whether denoising is performed first and then super-resolution or super-resolution is performed first and then denoising, the super-resolution effect will be unsatisfactory. Summary of the invention

[0004] In view of the shortcomings of the prior art described above, an object of the present invention is to provide a super-resolution method, medium and system based on machine learning, which is used to solve the problem in the prior art that whether denoising is performed first and then super-resolution or super-resolution is performed first and then denoising, the super-resolution effect is poor.

[0005] To achieve the above-mentioned purpose and other related purposes, the present invention provides a super-resolution method based on machine learning, and the super-resolution method based on machine learning includes: obtaining content statistical characteristics of video images; classifying the video images according to the content statistical characteristics of the video images to obtain at least one group; according to the group to which each video image belongs, in combination with the noise statistical characteristics of the video images, selecting a corresponding machine learning model for super-resolution reconstruction to obtain a corresponding high-resolution image.

[0006] In one embodiment of the present invention, according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image. An implementation method includes: performing noise estimation on the video image to obtain a corresponding noise level distribution map; performing channel splicing on the video image and the corresponding noise level distribution map to obtain a corresponding high-dimensional spliced ​​image; using the high-dimensional spliced ​​image as the input of the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

[0007] In one embodiment of the present invention, the machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes: obtaining multiple groups of data as first training data; each group of data includes a high-dimensional spliced ​​image corresponding to the low-resolution image with compressed noise belonging to the group and the corresponding high-resolution image without compressed noise; using the first training data to train a machine learning model to obtain the machine learning model corresponding to the group.

[0008] In one embodiment of the present invention, a method for classifying the video images according to content statistical characteristics and noise levels of the video images to obtain at least one group includes: obtaining signal characteristics and noise levels of the video images; classifying the video images according to content statistical characteristics to obtain at least one category of images; performing secondary classification of each category of images according to the noise level to obtain at least one subcategory; and classifying each subcategory again according to signal characteristics to obtain at least one group.

[0009] In one embodiment of the present invention, according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image. An implementation method includes: using the video image as the input of the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

[0010] In one embodiment of the present invention, the machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes: obtaining multiple groups of data as second training data; each group of data includes a low-resolution image with compressed noise belonging to the group and a corresponding high-resolution image without compressed noise; using the second training data to train a machine learning model to obtain the machine learning model corresponding to the group.

[0011] In one embodiment of the present invention, the machine learning model is a neural network model.

[0012] In one embodiment of the present invention, the machine learning model is a plurality of filter groups.

[0013] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the super-resolution method based on machine learning described in the present invention.

[0014] The present invention also provides a super-resolution system based on machine learning, and the super-resolution system based on machine learning includes: a noise estimation module, which is used to obtain a noise level distribution map of a video image; an image stitching module, which is connected to the noise estimation module and is used to perform channel stitching on the video image and the noise level distribution map to obtain a corresponding high-dimensional stitching image; a content statistical characteristic acquisition module, which is used to obtain the content statistical characteristics of the video image; a classification module, which is connected to the content statistical characteristic acquisition module and is used to classify the video image according to the content statistical characteristics of the video image to obtain at least one group; a model switching module, which is connected to the classification module and is used to select a corresponding machine learning model according to the group to which the video image belongs and in combination with the noise statistical characteristics of the video image; a machine learning module, which is respectively connected to the image stitching module and the model switching module and is used to call the corresponding machine learning model to perform super-resolution reconstruction on the video image to obtain a corresponding high-resolution image.

[0015] The present invention also provides a super-resolution system based on machine learning, characterized in that the super-resolution system based on machine learning includes: a noise estimation module, used to obtain the noise level of the video image; a content statistical characteristic acquisition module, used to obtain the content statistical characteristics of the video image; a signal feature acquisition module, used to obtain the signal characteristics of the video image; a classification module, connected to the noise estimation module, the content statistical characteristic acquisition module and the signal feature acquisition module respectively, used to classify the video image according to the noise level, content statistical characteristics and / or signal characteristics of the video image to obtain at least one group; a model switching module, connected to the classification module, used to select a corresponding machine learning model according to the group to which the video image belongs and in combination with the noise statistical characteristics of the video image; a machine learning module, connected to the model switching module, used to call the corresponding machine learning model to perform super-resolution reconstruction on the video image to obtain a corresponding high-resolution image.

[0016] As described above, the super-resolution method, medium and system based on machine learning of the present invention have the following beneficial effects:

[0017] In the super-resolution method based on machine learning, denoising and super-resolution are simultaneously achieved through a machine learning model, which effectively avoids the problem in the prior art that whether denoising is performed first and then super-resolution or super-resolution is performed first and then denoising, the super-resolution effect is not ideal;

[0018] In addition, the present invention combines the two steps of denoising and super-resolution into one step, which can realize the reuse of machine learning models. There is no need to train and test machine learning models for denoising and super-resolution separately, which reduces the amount of calculation. At the same time, it can achieve an adaptive balance between super-resolution reconstruction of adding high-frequency details and filtering out high-frequency ringing noise.

[0019] The super-resolution method based on machine learning includes estimating the compression noise level, and using the compression noise level distribution as the input of the machine learning model and the basis for classification, which not only improves the robustness of the video super-resolution effect under various picture qualities, but also avoids the problem of repeatedly loading multiple different model parameters during playback, reduces the storage space required for hardware, and improves playback efficiency;

[0020] The super-resolution method based on machine learning considers the statistical characteristics of the video, compression noise and signal characteristics as the classification basis to classify the video image, and selects the corresponding machine learning model according to the group to which the video image belongs. It has strong pertinence and high accuracy, and achieves better image super-resolution effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1Shown is a flow chart of a super-resolution method based on machine learning in one embodiment of the present invention.

[0022] Figure 2 FIG. 4 is a flow chart showing step S13 of the super-resolution method based on machine learning in one embodiment of the present invention.

[0023] Figure 3 Shown is a training flowchart of a machine learning model in one embodiment of the machine learning-based super-resolution method described in the present invention.

[0024] Figure 4 FIG. 4 is a flow chart of step S12 of the super-resolution method based on machine learning in one embodiment of the present invention.

[0025] Figure 5 Shown is a training flowchart of a machine learning model in one embodiment of the machine learning-based super-resolution method described in the present invention.

[0026] Fig. 6A Shown is a training flowchart of a neural network model in one embodiment of the super-resolution method based on machine learning described in the present invention.

[0027] Figure 6B Shown is a flowchart of super-resolution reconstruction of the super-resolution method based on machine learning in one embodiment of the present invention.

[0028] Fig. 7A Shown is a training flowchart of a neural network model in one embodiment of the super-resolution method based on machine learning described in the present invention.

[0029] Figure 7B Shown is a flowchart of super-resolution reconstruction of the super-resolution method based on machine learning in one embodiment of the present invention.

[0030] Fig. 8A FIG. 4 is a flowchart showing the training of a filter bank in an embodiment of the super-resolution method based on machine learning according to the present invention.

[0031] Figure 8B Shown is a flowchart of super-resolution reconstruction of the super-resolution method based on machine learning in one embodiment of the present invention.

[0032] Fig. 9 Shown is a structural diagram of a super-resolution system based on machine learning in one embodiment of the present invention.

[0033] Fig.10 Shown is a structural diagram of a super-resolution system based on machine learning in one embodiment of the present invention.

[0034] Component number description

[0035] 900 Super-resolution system based on machine learning

[0036] 910 Noise Estimation Module

[0037] 920 Image Stitching Module

[0038] 930 Content statistics feature acquisition module

[0039] 940 Classification Module

[0040] 950 Model Switching Module

[0041] 960 Machine Learning Modules

[0042] 1000 Super-resolution system based on machine learning

[0043] 1010 Noise Estimation Module

[0044] 1020 Content statistics feature acquisition module

[0045] 1030 Signal feature acquisition module

[0046] 1040 Classification Module

[0047] 1050 Model Switching Module

[0048] 1060 Machine Learning Module

[0049] Steps S11 to S13

[0050] Steps S131 to S133

[0051] Steps S31 to S32

[0052] Steps S121 to S124

[0053] Steps S51 to S52

[0054] Steps S61a to S64a

[0055] Steps S61b to S65b

[0056] Steps S71a to S74a

[0057] Steps S71b to S74b

[0058] Steps S81a to S83a

[0059] Steps S81b to S84b DETAILED DESCRIPTION

[0060] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0061] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0062] Super-resolution refers to improving the resolution of the original image by hardware or software methods. The process of super-resolution is the process of adding high-frequency details to the original low-resolution image. Low-resolution compressed videos generally have image quality problems such as blurred details, block effects, ringing noise, and mosquito noise. If super-resolution and detail enhancement are performed directly on the picture, while supplementing the high-frequency details lost due to amplification, the noise may also be supplemented as details, making it impossible to filter out ideally through the subsequent denoising module. If denoising is performed first, some details are filtered out as noise, making it difficult for subsequent super-resolution to restore more details. Therefore, whether denoising is performed first and then super-resolution or super-resolution is performed first and then denoising, the super-resolution effect will be unsatisfactory. In addition, Internet videos usually use a constant bitrate (CBR) transcoding method, and macroblocks of different picture complexities may use different QPs (Quantization Parameters), which will result in various compression noises of different intensities in the same video sequence. Therefore, using one or more global bitrate parameters cannot fully simulate the compression of actual online videos.

[0063] In view of the above problems, the present invention provides a super-resolution method based on machine learning, which includes: obtaining content statistical characteristics and noise levels of video images; classifying the video images according to the content statistical characteristics and noise levels of the video images to obtain at least one group; selecting a corresponding machine learning model for super-resolution reconstruction according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image to obtain a corresponding high-resolution image. In the super-resolution method based on machine learning, a corresponding machine learning model is selected for super-resolution reconstruction according to the group to which the video image belongs, and denoising and super-resolution are simultaneously realized through a neural network, thereby avoiding the problem in the prior art that whether denoising is performed first and then super-resolution or super-resolution is performed first and then denoising, the super-resolution effect is poor.

[0064] See also Figure 1 In one embodiment of the present invention, the super-resolution method based on machine learning includes:

[0065] S11, obtaining content statistical characteristics of a video image, wherein the video image refers to an image corresponding to a part or all of the frames in a video, which is a low-resolution image with compression noise.

[0066] S12: Classify the video image according to content statistical characteristics of the video image to obtain at least one group.

[0067] S13, selecting a corresponding machine learning model for super-resolution reconstruction according to the group to which each video image belongs and combining the noise statistical characteristics of the video image to obtain a corresponding super-resolution reconstructed image. The super-resolution reconstructed image is a high-resolution image without compression noise.

[0068] Machine Learning (ML) is a multi-disciplinary subject that mainly studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Common machine learning models include linear models, kernel methods and support vector machines, decision tree models, neural network models, filter banks, etc. Within machine learning, the entire machine learning establishment and operation mechanism includes the following steps: data collection, data preparation (including dividing data into training data and evaluation data), model selection, model training, model evaluation, parameter fine-tuning, and practical application. Model selection and training are crucial links in the entire machine learning process.

[0069] See also Figure 2In one embodiment of the present invention, according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image. An implementation method includes:

[0070] S131, perform noise estimation on the video image to obtain a corresponding noise level distribution map; the noise mainly refers to compression noise. The noise level of each pixel in the video image is obtained by performing noise estimation on each pixel, and the noise level distribution map corresponding to the video image can be obtained according to the noise level of each pixel. In practical applications, the video image can be divided into multiple n×n coding blocks, and noise estimation is performed on each coding block to obtain the noise level corresponding to different coding blocks; the noise level distribution map corresponding to the entire video image can be obtained according to the noise level corresponding to each coding block. Among them, the value of n is determined by the video coding standard, for example: 8, 16, 32.

[0071] The noise level distribution map is a grayscale map. The grayscale values ​​of the pixels on the grayscale map are quantized. Therefore, the grayscale value space of each point on the noise level distribution map is limited. For example, the grayscale value space corresponding to 64 noise levels is {0, 4, 8, ..., 252}. The grayscale value of each pixel in the noise level distribution map depends on the noise level of the point.

[0072] The image can be stored as a three-dimensional matrix, the three dimensions of which are (width, height, channel), where width is the width of the image, height is the height of the image, and channel is the channel of the image. In this embodiment, for the video image, channel=3, corresponding to the R channel, G channel, and B channel respectively; the noise level distribution map is a grayscale map, and its channel=1. S132, channel-joining the video image and the corresponding noise level distribution map to obtain a corresponding high-dimensional high-dimensional jointed image.

[0073] Traditional methods generally only analyze the spatial information of noisy images. This embodiment measures the video image from two dimensions: the spatial information of the original image and the auxiliary information of the noise level, wherein the auxiliary information of the noise level is represented by the grayscale of the pixel points in the noise level distribution map. This embodiment obtains a corresponding high-dimensional spliced ​​image by channel splicing the video image with the noise level distribution map of the same resolution. The high-dimensional spliced ​​image carries the original pixels of the video image and the noise level information. As described above, both the video image and the noise level image can be stored in a three-dimensional matrix. For video images and noise level images with the same height and width, the two can be spliced ​​in the channel dimension to achieve channel splicing and obtain the corresponding high-dimensional spliced ​​image. S133, the high-dimensional spliced ​​image and the video image are used as the input of the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

[0074] In this embodiment, different groups correspond to different machine learning models. To achieve super-resolution reconstruction, a machine learning model corresponding to the group should be selected according to the group to which the video image belongs, and the high-dimensional spliced ​​image and the video image should be processed using the machine learning model to obtain a noise-free high-resolution image corresponding to the video image.

[0075] In this embodiment, the noise level distribution map and the video image are channel-joined in the third dimension to obtain a corresponding high-dimensional stitched image, so that the high-dimensional stitched image can fully reflect the noise level of different macroblocks, thereby simulating the compression of actual online videos.

[0076] In one embodiment of the present invention, channel=3 in the three-dimensional matrix corresponding to the video image, that is, the video image includes three channels of R / G / B. The noise level distribution map is a grayscale map, which can be represented by only one channel value, and its corresponding three-dimensional matrix has channel=1, that is, the noise level distribution map only contains one noise channel. A four-channel image can be obtained by splicing the R channel, G channel, B channel of the video image and one channel of the noise level distribution map, and the three-dimensional matrix corresponding to the four-channel image has channel=4, that is, the four-channel image contains the R channel, G channel, B channel and noise channel.

[0077] In another embodiment of the present invention, the video image is transferred to the Y-Cb-Cr space. Since the human eye is more sensitive to the brightness information Y than the chrominance information Cb / Cr, in order to improve the operation efficiency, only the Y channel can be magnified based on machine learning, and Cb / Cr adopts a fast magnification algorithm based on traditional interpolation. At this time, the video image input to the model has only one brightness channel, and correspondingly, its channel=1. After channel splicing of the brightness channel of the video image and one channel of the noise level distribution map, a two-channel image can be obtained. The three-dimensional matrix corresponding to the two-channel image has channel=2, that is, the two-channel image contains a brightness channel and a noise channel.

[0078] In one embodiment of the present invention, the noise level distribution map is obtained by a quantization parameter map (QP map) output by a decoder. The larger the quantization parameter, the larger the corresponding compression noise. Therefore, the compression noise can be evaluated accordingly according to the size of the quantization parameter displayed in the quantization parameter map. The quantization parameter map can be directly obtained using an existing decoder, and the specific method is not repeated here.

[0079] In one embodiment of the present invention, the noise level distribution diagram is obtained by statistically analyzing local features of the decoded image. For example, the compression noise level can be estimated by using information such as the mean, variance, and dynamic range in a local window or coding block.

[0080] In one embodiment of the present invention, the noise level distribution map is obtained by machine learning, for example, by using images with different noise levels and their corresponding noise levels as training data to train another neural network, and using the trained neural network to achieve compression noise estimation and image quality prediction.

[0081] See also Figure 3 In one embodiment of the present invention, the machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes:

[0082] S31, obtaining multiple groups of data as first training data; for any group of data, it includes a high-dimensional spliced ​​image corresponding to a low-resolution image with compression noise belonging to the group, and a high-resolution image without compression noise corresponding to the low-resolution image with compression noise. The low-resolution image and its corresponding high-resolution image can be obtained from an external database, and the corresponding high-dimensional spliced ​​image can be obtained through step S132.

[0083] In this embodiment, each group corresponds to a set of first training data. For any group, the content statistical characteristics and noise levels of all low-resolution images in the corresponding first training data match the group, that is, all low-resolution images in the first training data belong to the group according to the grouping method of step S11 and step S12.

[0084] S32: Use the first training data to train a machine learning model to obtain a machine learning model corresponding to the grouping.

[0085] In this embodiment, each group corresponds to a set of first training data and a machine learning model. The machine learning model corresponding to the group can be obtained by training a machine learning model using the first training data. After the training is completed, the machine learning model corresponding to each group can simultaneously achieve super-resolution reconstruction and compression noise removal for the low-resolution image in the group, avoiding the problem in the prior art that whether denoising first and then performing super-resolution or first performing super-resolution and then denoising will result in poor super-resolution effect.

[0086] In one embodiment of the present invention, according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image. An implementation method includes: performing noise estimation on the video image to obtain a corresponding noise level distribution map; using the video image and its corresponding noise level distribution map as input to the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

[0087] In this embodiment, the machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes: obtaining multiple groups of data as third training data; each group of data includes a low-resolution image with compression noise belonging to the group, a noise level distribution map corresponding to the low-resolution image with compression noise, and a corresponding high-resolution image without compression noise; using the third training data to train a machine learning model to obtain the machine learning model corresponding to the group. The low-resolution image and its corresponding high-resolution image can be obtained from an external database, and the corresponding noise level distribution map can be obtained by step S131.

[0088] See also Figure 4 In one embodiment of the present invention, a method for classifying the video image according to the content statistical characteristics of the video image to obtain at least one group includes:

[0089] S121, acquiring signal characteristics and noise levels of the video image; the signal characteristics of the video image, such as spatial local characteristics of pixel points, are used to reflect local characteristics of the video image;

[0090] S122: Classify the video images according to content statistical characteristics to obtain at least one category of images.

[0091] Due to the diversity of video content, it is necessary to pre-classify the video according to the statistical characteristics of the content without increasing the complexity of the model. In this embodiment, the content statistical characteristics can be the values ​​of the following parameters: the amount of information, grayscale average, grayscale mode, grayscale median, gradient and / or variance of the image, or the maximum value and / or average value of the above parameters. According to the above statistical characteristics, the video image can be divided into multiple categories. For example, since the computer graphic image (Computer Graphic Image, CGI) has more noise-free flat areas, sharper text and edges than the natural scene image (Natural Scene Image, NSI); the flatness of the image area can be characterized by the maximum and average value of the variance, and the sharpness of the text and the edge characteristics can be characterized by the gradient; therefore, according to the variance and gradient characteristics of the video image, the video image can be divided into two categories: natural scene image and computer graphic image. In practical applications, the video image can be divided into categories such as buildings, faces, and natural scenery according to the specific content statistical characteristics of the video.

[0092] In particular, if the content statistical characteristics of all video images are the same or have little difference, the video images can be divided into one type of images according to the content statistical characteristics according to actual needs.

[0093] S123, performing secondary classification on each type of image according to the noise level to obtain at least one subclass;

[0094] The information loss caused by video compression can easily produce block effects, ringing noise, mosquito noise, etc. in the image. In this embodiment, the noise level of the video image is used as the basis for the grouping process. Specifically, the classified video image is reclassified according to the noise level of the video image to obtain at least one group. For example, if step S122 divides the video image into two types of images, CGI and NSI, then in step S123, the CGI image and the NSI image are respectively secondary classified based on the noise level to obtain four sub-classes of high-noise CGI image, low-noise CGI image, high-noise NSI image and low-noise NSI image.

[0095] In particular, when the noise levels of all video images contained in a class of images are not much different, all images in this class of images can be regarded as a subclass according to actual needs.

[0096] Preferably, each type of image is divided into the same number of sub-categories.

[0097] S124: reclassify each subclass according to the signal feature to obtain at least one group.

[0098] In this step, by performing spatial or frequency domain analysis on the video image, signal features are extracted to facilitate reclassification of the images in each subclass. For example, by judging the local gradient characteristics, brightness variance, bandwidth, etc. of the pixels, the pixels can be divided into groups such as flat, weak texture, and edge; the edge direction information of the image can also be obtained by analyzing the signal features. For another example, if the image is divided into four subclasses of high-noise CGI images, low-noise CGI images, high-noise NSI images, and low-noise NSI images in step S123, then in this step, each subclass can be reclassified according to the signal features to obtain eight groups of flat high-noise CGI images, weak-texture high-noise CGI images, flat low-noise CGI images, weak-texture low-noise CGI images, flat high-noise NSI images, weak-texture high-noise NSI images, flat low-noise NSI images, and weak-texture low-noise NSI images.

[0099] Preferably, each subcategory is divided into groups of equal number.

[0100] In one embodiment of the present invention, according to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image. An implementation method includes: using the video image as the input of the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

[0101] See also Figure 5 In one embodiment of the present invention, the machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes:

[0102] S51, obtain multiple groups of data as second training data; each group of data includes a low-resolution image with compression noise belonging to the group and a corresponding high-resolution image without compression noise. The low-resolution image and its corresponding high-resolution image can be obtained from an external database, and each group of data contains a "LR image-HR image" mapping relationship. In this embodiment, each group corresponds to a group of second training data. For any group, the content statistical characteristics and noise levels of all LR images in the corresponding second training data match the group, that is, all LR images in the second training data belong to the group according to the grouping method of step S11 and step S12.

[0103] S52: Use the second training data to train a machine learning model to obtain a machine learning model corresponding to the grouping.

[0104] In this embodiment, each group corresponds to a machine learning model. The machine learning model corresponding to the group can be obtained by training a machine learning model using the second training data. After the training is completed, the machine learning model corresponding to each group can simultaneously achieve super-resolution reconstruction and compression noise removal for the low-resolution image in the group, avoiding the problem in the prior art that whether denoising first and then performing super-resolution or first performing super-resolution and then denoising will result in poor super-resolution effect.

[0105] In one embodiment of the present invention, the machine learning model is a neural network model.

[0106] A neural network is a computational model that consists of a large number of interconnected nodes (or neurons). Each node represents a specific output function, called an activation function. The connection between each two nodes represents a weighted value for the signal passing through the connection, called a weight. The output of the network varies depending on the network connection method, weight value, and activation function. A neural network can form a complete network structure through proper training, and then any input can obtain the corresponding output through the network structure.

[0107] Preferably, the neural network model used in this embodiment is CARN (Cascading Residual Network) or RCAN (Residual Channel Attention Networks).

[0108] See also Fig. 6A In one embodiment of the present invention, the step of training the neural network includes:

[0109] S61a, obtaining content statistical characteristics of the video image and classifying the video image according to the content statistical characteristics to obtain multiple groups; the content statistical characteristics may be gradient, variance, etc.;

[0110] S62a, performing noise evaluation on the video image to obtain a corresponding noise level distribution map; the noise level distribution map can be obtained by the QP output by the decoder, or by counting local features of the decoded image, or by a neural network;

[0111] S63a, performing channel stitching on the noise level distribution map and the video image to obtain a corresponding high-dimensional stitching image;

[0112] S64a, for each group, a neural network model is trained using the high-dimensional spliced ​​image and the corresponding high-resolution image to obtain a neural network model corresponding to the group. The neural network training can be obtained by an existing training method, for example, by implementing the training of the neural network through a back propagation training model.

[0113] See also Figure 6B In this embodiment, the method for implementing super-resolution reconstruction for any LR image with compression noise includes:

[0114] S61b, obtaining content statistical characteristics of the LR image and determining the group to which the LR image belongs according to the content statistical characteristics;

[0115] S62b, performing noise evaluation on the LR image and obtaining a corresponding noise level distribution map;

[0116] S63b, performing channel stitching on the noise level distribution map and the LR image to obtain a corresponding high-dimensional stitching image;

[0117] S64b, selecting a corresponding neural network model according to the group to which the LR image belongs;

[0118] S65b, using the high-dimensional spliced ​​image corresponding to the LR image as the input of the neural network model selected in step S64b, and the output of the neural network model is the high-resolution image corresponding to the LR image.

[0119] Preferably, the neural network is CARN, RCAN, etc. Experiments show that by using suitable training data, the neural network structure can not only optimize the super-resolution effect but also remove compression noise at the same time. The number of neural networks used in the super-resolution method based on machine learning in this embodiment is N, where N is the number of groups obtained by classifying the video image according to the content statistical characteristics.

[0120] See also Fig. 7A In one embodiment of the present invention, the step of training the neural network includes:

[0121] S71a, obtaining content statistical characteristics, noise level, and signal characteristics of the video image;

[0122] S72a, classifying the video image according to content statistical characteristics, noise level, and signal characteristics of the video image to obtain at least one group;

[0123] S73a, for each group, a neural network model is trained using the video image in the group and its corresponding high-resolution image to obtain the neural network model corresponding to the group. The training of the neural network can be obtained by an existing training method, for example, the training of the neural network is realized by a back propagation training model, and the video image and its corresponding high-resolution image can be obtained from an external database.

[0124] See also Figure 7B In this embodiment, the method for implementing super-resolution reconstruction for any LR image with compression noise includes:

[0125] S71b, obtaining content statistical characteristics, noise level and signal characteristics of the LR image; the noise level can be characterized by a noise level, for example, the noise can be divided into levels 0-8;

[0126] S72b, determining the group to which the LR image belongs according to content statistical characteristics, noise level, and signal characteristics of the LR image;

[0127] S73b, selecting a corresponding neural network model according to the group to which the LR image belongs;

[0128] S74b, using the LR image as the input of the neural network model selected in step S73b, and the output of the neural network model is the high-resolution image corresponding to the LR image.

[0129] In this embodiment, there is no need to splice the LR image and the noise level distribution map in the third dimension, which reduces the operation of image splicing processing. However, in this embodiment, the LR image needs to be classified according to the content statistical characteristics, noise level and signal characteristics of the LR image, and more groups are obtained, each group corresponds to a neural network model, which increases the algorithm complexity in this embodiment and the operation speed is relatively slow.

[0130] In this embodiment, the classification of video images comprehensively considers the global characteristics, noise levels and local characteristics (pixel-level characteristics) of the video images. Therefore, the classification results can fully reflect the compression noise level of each macroblock, thereby simulating the compression situation of actual online videos.

[0131] In one embodiment of the present invention, the machine learning model is a plurality of filter groups.

[0132] See also Fig. 8A In one embodiment of the present invention, the step of training the filter bank comprises:

[0133] S81a, obtaining content statistical characteristics, noise level, and signal characteristics of the video image; the noise level can be characterized by a noise level, for example, the noise can be divided into levels 0-8;

[0134] S82a, classifying the video image according to content statistical characteristics, noise level, and signal characteristics of the video image to obtain at least one group;

[0135] S83a: For each group, use the video images in the group and their corresponding high-resolution images to train multiple filter groups to obtain a filter group model corresponding to the group.

[0136] See also Figure 8B In this embodiment, the method for implementing super-resolution reconstruction for any LR image with compression noise includes:

[0137] S81b, obtaining content statistical characteristics, noise level and signal characteristics of the LR image;

[0138] S82b, determining the group to which the LR image belongs according to content statistical characteristics, noise level, and signal characteristics of the LR image;

[0139] S83b, selecting a corresponding filter bank model according to the group to which the LR image belongs;

[0140] S84b, using the LR image as the input of the filter bank model selected in step S83b, and the output of the filter bank model is the high-resolution image corresponding to the LR image.

[0141] Compared with the neural network model, the filter bank model structure is simpler and the training and operation speeds are faster. However, since the content statistical characteristics, noise level and signal characteristics are comprehensively considered when classifying the video image in the present embodiment, the classification is finer and the grouping obtained after the classification is more, so more filter groups need to be stored in the present embodiment. In addition, since the local characteristics (noise level and signal characteristics) of the video image are considered when grouping in the present embodiment, adjacent pixels in the same video image may be located in different groups, resulting in adjacent pixels may need to adopt different filter models. When adjacent pixels adopt different filter models, it is necessary to switch between different models, and this switching may cause discontinuity of the picture.

[0142] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the super-resolution method based on machine learning described in the present invention.

[0143] See also Fig. 9The present invention further provides a super-resolution system based on machine learning, wherein the super-resolution system based on machine learning 900 comprises:

[0144] The noise estimation module 910 is used to implement step S131, that is, to perform noise estimation on the video image and obtain a corresponding noise level distribution map;

[0145] The image stitching module 920 is connected to the noise estimation module 910 and is used to implement step S132, namely: performing channel stitching on the video image and the corresponding noise level distribution map to obtain a corresponding high-dimensional stitching image;

[0146] The content statistical characteristic acquisition module 930 is used to implement step S11, namely: acquiring the content statistical characteristic of the video image;

[0147] A classification module 940, connected to the content statistical characteristic acquisition module 930, is used to classify the video image according to the content statistical characteristic of the video image to obtain at least one group;

[0148] A model switching module 950, connected to the classification module 940, is used to select a corresponding machine learning model according to the group to which the video image belongs and in combination with the noise statistical characteristics of the video image in step S13;

[0149] The machine learning module 960 is connected to the image stitching module 920 and the model switching module 950 respectively, and is used to call the corresponding machine learning model to implement the super-resolution reconstruction of the video image in step S13 to obtain the corresponding high-resolution image.

[0150] See also Fig.10 The present invention further provides a super-resolution system based on machine learning, characterized in that the super-resolution system based on machine learning 1000 comprises:

[0151] The noise estimation module 1010 is used to obtain the noise level of the video image in step S121;

[0152] The content statistical characteristic acquisition module 1020 is used to implement step S11, namely: acquiring the content statistical characteristics of the video image;

[0153] A signal feature acquisition module 1030, used for acquiring the signal feature of the video image in S121;

[0154] A classification module 1040, connected to the noise estimation module 1010, the content statistical characteristic acquisition module 1020 and the signal feature acquisition module 1030 respectively, for implementing step S12, classifying the video image according to the noise level, content statistical characteristics and / or signal characteristics of the video image to obtain at least one group;

[0155] A model switching module 1050, connected to the classification module 1040, is used to select a corresponding machine learning model according to the group to which the video image belongs and in combination with the noise statistical characteristics of the video image in step S13;

[0156] The machine learning module 1060 is connected to the model switching module 1050, and is used to call the corresponding machine learning model to implement super-resolution reconstruction of the video image in step S13 to obtain a corresponding high-resolution image.

[0157] The protection scope of the super-resolution method based on machine learning described in the present invention is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present invention are included in the protection scope of the present invention.

[0158] The present invention also provides a super-resolution method system based on machine learning, and the super-resolution system based on machine learning can implement the super-resolution method based on machine learning described in the present invention. However, the implementation device of the super-resolution method based on machine learning described in the present invention includes but is not limited to the structure of the super-resolution system based on machine learning listed in this embodiment. All structural deformations and replacements of the prior art made according to the principles of the present invention are included in the protection scope of the present invention.

[0159] In the present invention, by adopting a machine learning model to combine the two steps of super-resolution reconstruction and decompression noise together, there is no need to train, learn and test the super-resolution reconstruction and decompression noise separately, which can realize the reuse of the neural network structure or filter group and reduce the amount of calculation;

[0160] In the present invention, super-resolution reconstruction and decompression noise can be achieved simultaneously, and the image quality obtained is significantly improved compared with the traditional super-resolution method. It not only eliminates the aliasing and edge jaggedness introduced by image amplification, but also suppresses the ringing noise caused by video compression quantization, and reduces the noise deterioration caused by image amplification.

[0161] In the present invention, super-resolution reconstruction and decompression noise are simultaneously achieved by using machine learning, and an adaptive balance can be achieved between super-resolution reconstruction of adding high-frequency details and filtering out high-frequency ringing noise by adjusting the corresponding training data;

[0162] In the present invention, video images are classified according to content statistical characteristics based on the statistical characteristics of the images, and the corresponding machine learning model is selected according to the classification results. By establishing machine learning models for video images with different contents and performing super-resolution reconstruction, the output noise-free HR image has higher quality.

[0163] In the present invention, the compression noise level and local signal characteristics of the video images are also considered when classifying the video images, and the weights of the neural network or the coefficients of the filter group are determined by AI learning, which not only improves the robustness of the video super-resolution under various image qualities, but also avoids the problem of repeatedly loading multiple different network models or filter group models during playback, reduces the storage space required for the hardware, and improves the playback efficiency;

[0164] In the present invention, three methods can be used to estimate compression noise, including: obtaining through the QP map output by the decoder; obtaining through statistical local features of the decoded image; and using neural networks to achieve compression noise estimation and image quality prediction by learning training images with different noise levels. In actual application, the corresponding compression noise estimation method can be selected according to the specific situation, which is highly flexible.

[0165] In summary, the present invention effectively overcomes various shortcomings of the prior art and has high industrial utilization value.

[0166] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A super-resolution method based on machine learning, characterized in that: The super-resolution method based on machine learning includes: Obtaining content statistical characteristics of video images; Classifying the video image according to content statistical characteristics of the video image to obtain at least one group; According to the group to which each video image belongs and in combination with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image, which includes: estimating the noise of the video image, obtaining a corresponding noise level distribution map, channel-splicing the video image and the corresponding noise level distribution map to obtain a corresponding high-dimensional spliced ​​image, and using the high-dimensional spliced ​​image as the input of the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

2. The super-resolution method based on machine learning according to claim 1, characterized in that: The machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes: Acquire multiple groups of data as first training data; each group of data includes a high-dimensional spliced ​​image corresponding to the low-resolution image with compression noise belonging to the group and a corresponding high-resolution image without compression noise; A machine learning model is trained using the first training data to obtain a machine learning model corresponding to the grouping.

3. The super-resolution method based on machine learning according to claim 1, characterized in that: An implementation method for classifying the video image according to content statistical characteristics of the video image to obtain at least one group includes: Acquiring signal characteristics and noise levels of the video image; Classifying the video images according to content statistical characteristics to obtain at least one category of images; Perform secondary classification on each type of image according to the noise level to obtain at least one sub-class; Each subclass is reclassified according to the signal feature to obtain at least one group.

4. The super-resolution method based on machine learning according to claim 3, characterized in that: According to the group to which each video image belongs, combined with the noise statistical characteristics of the video image, a corresponding machine learning model is selected for super-resolution reconstruction to obtain a corresponding high-resolution image. One implementation method includes: The video image is used as the input of the corresponding machine learning model, and the output of the corresponding machine learning model is the corresponding high-resolution image.

5. The super-resolution method based on machine learning according to claim 4, characterized in that: The machine learning model corresponding to each group is obtained through training; the training method of the machine learning model includes: Acquire multiple groups of data as second training data; each group of data includes a low-resolution image with compression noise belonging to the group and a corresponding high-resolution image without compression noise; A machine learning model is trained using the second training data to obtain a machine learning model corresponding to the grouping.

6. The super-resolution method based on machine learning according to any one of claims 1 to 5, characterized in that: The machine learning model is a neural network model.

7. The super-resolution method based on machine learning according to any one of claims 3 to 5, characterized in that: The machine learning model is a plurality of filter groups.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the super-resolution method based on machine learning described in any one of claims 1 to 7 is implemented.

9. A super-resolution system based on machine learning, characterized in that: The machine learning-based super-resolution system includes: A noise estimation module, used to obtain a noise level distribution map of a video image; An image stitching module, connected to the noise estimation module, for performing channel stitching on the video image and the noise level distribution map to obtain a corresponding high-dimensional stitching image; A content statistical characteristic acquisition module, used to acquire content statistical characteristics of the video image; A classification module, connected to the content statistical characteristic acquisition module, used to classify the video image according to the content statistical characteristics of the video image to obtain at least one group; A model switching module, connected to the classification module, for selecting a corresponding machine learning model according to the group to which the video image belongs and in combination with the noise statistical characteristics of the video image; The machine learning module is connected to the image stitching module and the model switching module respectively, and is used to call the corresponding machine learning model to perform super-resolution reconstruction on the video image to obtain a corresponding high-resolution image.

10. A super-resolution system based on machine learning, characterized in that: The machine learning-based super-resolution system includes: A noise estimation module, used for obtaining the noise level of the video image; A content statistical characteristic acquisition module, used to acquire content statistical characteristics of the video image; A signal feature acquisition module, used to acquire the signal features of the video image; a classification module, connected to the noise estimation module, the content statistical characteristic acquisition module and the signal feature acquisition module respectively, and configured to classify the video image according to the noise level, content statistical characteristic and signal characteristic of the video image to obtain at least one group; A model switching module, connected to the classification module, for selecting a corresponding machine learning model according to the group to which the video image belongs and in combination with the noise statistical characteristics of the video image; The machine learning module is connected to the model switching module and is used to call the corresponding machine learning model to perform super-resolution reconstruction on the video image to obtain a corresponding high-resolution image.

Images