Real-time video super-resolution

Abstract

A computer-implemented method for increasing the image resolution of a digital image. The method includes performing bicubic upsampling on the digital image to generate a base high-resolution (HR) image. The digital image is converted from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. Multiple convolutional layers of a neural network model are applied to the LR residual image to convert the LR residual image into multiple HR residual sub-images corresponding to the digital image. An HR image corresponding to the digital image is generated using the base HR image and the multiple HR residual sub-images.

Description

Technical Field

[0001] The present invention relates to video data processing in a computing device. Background Art

[0002] With the rapid advancement of electronic technology, 2K and 4K high-definition (HD) displays have become standard for home theater systems, tablets, and mobile devices. Videos played on mobile devices are often lower resolution due to limitations associated with video capture devices, storage space, network bandwidth, and / or data streaming. Therefore, before displaying individual frames on modern devices, it is important to increase the effective resolution and visual quality of videos in order to fully utilize the new generation of HD displays.

[0003] Existing image resolution enhancement techniques are primarily designed to produce high-quality images free of noise and artifacts. When noise and blocking artifacts are present in the input video (e.g., videos viewed on mobile devices), using existing techniques often leads to further deterioration in image quality. Furthermore, due to computational complexity, current image resolution enhancement techniques do not improve peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to traditional interpolation techniques, limiting their applicability for image processing on mobile devices with HD displays. Summary of the Invention

[0004] Various examples are now described to briefly introduce some concepts that will be further described in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] According to a first aspect of the present invention, a computer-implemented method for increasing the image resolution of a digital image is provided. The method comprises performing bicubic upsampling on the digital image to generate a base high-resolution (HR) image. The digital image is converted from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. The LR residual image is converted into a plurality of HR residual sub-images corresponding to the digital image using a plurality of convolutional layers of a neural network model. An HR image corresponding to the digital image is generated using the base HR image and the plurality of HR residual sub-images.

[0006] According to the first aspect, in a first implementation of the method, pixel shifting is performed on the multiple HR residual sub-images to generate an HR residual image. Generating the HR image corresponding to the digital image includes combining the HR residual image and the base HR image.

[0007] According to the first aspect or any of the above implementations of the first aspect, in a second implementation of the method, the neural network model includes an input layer, and the multiple convolutional layers include four convolutional layers.

[0008] According to the first aspect or any of the above implementations of the first aspect, in a third implementation of the method, the input layer is used to receive the digital image, and the output layers of the four convolutional layers are used to output the multiple HR residual sub-images.

[0009] According to the first aspect or any above-mentioned implementation manner of the first aspect, in a fourth implementation manner of the method, the first layer among the multiple convolutional layers is configured with a 3×3 pixel kernel and 8 channels; the second layer among the multiple convolutional layers is configured with a 3×3 pixel kernel and 6 channels; the third layer among the multiple convolutional layers is configured with a 3×3 pixel kernel and 4 channels; and the fourth layer among the multiple convolutional layers is configured with 4 channels.

[0010] According to the first aspect or any of the aforementioned implementations of the first aspect, in a fifth implementation of the method, the neural network model is trained using a plurality of training image pairs. Each of the plurality of training image pairs includes: an LR image corresponding to the training image; and a plurality of HR residual sub-images corresponding to the training image and configured as target outputs of the neural network model. The LR images have degraded image quality and are configured as inputs to the neural network model.

[0011] According to the first aspect or any of the above-mentioned implementations of the first aspect, in a sixth implementation of the method, training the neural network model includes: filtering the training image using a low-pass filter to generate a filtered image; downsampling the filtered image to generate a downsampled LR image; and degrading the image quality of the downsampled LR image by adding noise and artifacts to generate the LR image corresponding to the training image.

[0012] According to the first aspect or any above-mentioned implementation manner of the first aspect, in a seventh implementation manner of the method, training the neural network model includes: applying an unbalanced unsharp mask to the training image to generate a contrast-enhanced image; subtracting an upsampled version of the downsampled LR image from the contrast-enhanced image to generate an HR residual image corresponding to the training image.

[0013] According to the first aspect or any of the above-mentioned implementations of the first aspect, in an eighth implementation of the method, training the neural network model also includes: dividing the HR residual image corresponding to the training image to generate the multiple HR residual sub-images corresponding to the training image.

[0014] According to a second aspect of the present invention, a system for improving the image resolution of a digital image is provided. The system includes a memory storing instructions and one or more processors communicating with the memory. The one or more processors execute the instructions to perform bicubic upsampling on the digital image to generate a base high-resolution (HR) image. The digital image is converted from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. Multiple convolutional layers of a neural network model are applied to the LR residual image to convert the LR residual image into multiple HR residual sub-images corresponding to the digital image. An HR image corresponding to the digital image is generated using the base HR image and the multiple HR residual sub-images.

[0015] According to the second aspect, in a first implementation of the system, the one or more processors are further used to perform pixel shifting on the multiple HR residual sub-images to generate an HR residual image, and combine the HR residual image and the basic HR image to generate the HR image corresponding to the digital image.

[0016] According to the second aspect or any of the above implementations of the second aspect, in the second implementation of the system, the neural network model includes an input layer, and the multiple convolutional layers include four convolutional layers. The input layer is used to receive the digital image. The output layers of the four convolutional layers are used to output the multiple HR residual sub-images.

[0017] According to the second aspect or any of the above implementations of the second aspect, in a third implementation of the system, the first layer of the multiple convolutional layers is configured with a 3×3 pixel kernel and 8 channels; the second layer of the multiple convolutional layers is configured with a 3×3 pixel kernel and 6 channels; the third layer of the multiple convolutional layers is configured with a 3×3 pixel kernel and 4 channels; and the fourth layer of the multiple convolutional layers is configured with 4 channels.

[0018] According to the second aspect or any of the above implementations of the second aspect, in a fourth implementation of the system, the one or more processors are further configured to train the neural network model using multiple training image pairs. Each of the multiple training image pairs includes: an LR image corresponding to the training image; and multiple HR residual sub-images corresponding to the training image and configured as target outputs of the neural network model. The LR images have degraded image quality and are configured as inputs to the neural network model.

[0019] According to the second aspect or any of the above-mentioned implementations of the second aspect, in a fifth implementation of the system, the one or more processors are further used to: filter the training image using a low-pass filter to generate a filtered image; downsample the filtered image to generate a downsampled LR image; and degrade the image quality of the downsampled LR image by adding noise and artifacts to generate the LR image corresponding to the training image.

[0020] According to the second aspect or any of the above-mentioned implementations of the second aspect, in a sixth implementation of the system, in order to train the neural network model, the one or more processors are further used to: apply an unbalanced unsharp mask to the training image to generate a contrast-enhanced image; and subtract an upsampled version of the downsampled LR image from the contrast-enhanced image to generate an HR residual image corresponding to the training image.

[0021] According to the second aspect or any of the above-mentioned implementations of the second aspect, in a seventh implementation of the system, in order to train the neural network model, the one or more processors are further used to: divide the HR residual image corresponding to the training image to generate the multiple HR residual sub-images corresponding to the training image.

[0022] According to a third aspect of the present invention, a non-transitory computer-readable medium is provided, storing instructions for increasing the image resolution of a digital image. When executed by one or more processors, the instructions cause the one or more processors to perform operations, the operations comprising performing bicubic upsampling on the digital image to generate a base high-resolution (HR) image. The digital image is converted from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. Multiple convolutional layers of a neural network model are applied to the LR residual image to convert the LR residual image into multiple HR residual sub-images corresponding to the digital image. An HR image corresponding to the digital image is generated using the base HR image and the multiple HR residual sub-images.

[0023] According to the third aspect, in a first implementation of the non-transitory computer-readable medium, the operations further include training the neural network model using a plurality of training image pairs. Each of the plurality of training image pairs includes: a LR image corresponding to the training image; and a plurality of HR residual sub-images corresponding to the training image and configured as target outputs of the neural network model. The LR images have degraded image quality and are configured as inputs to the neural network model.

[0024] According to the third aspect or any of the above implementations of the third aspect, in a second implementation of the non-transitory computer-readable medium, the operation further includes: filtering the training image using a low-pass filter to generate a filtered image; downsampling the filtered image to generate a downsampled LR image; and degrading the image quality of the downsampled LR image by adding noise and artifacts to generate the LR image corresponding to the training image.

[0025] According to the third aspect or any above-mentioned implementation manner of the third aspect, in the third implementation manner of the non-transitory computer-readable medium, the operation also includes: applying an unbalanced unsharp mask to the training image to generate a contrast-enhanced image; subtracting an upsampled version of the downsampled LR image from the contrast-enhanced image to generate an HR residual image corresponding to the training image; dividing the HR residual image corresponding to the training image to generate the multiple HR residual sub-images corresponding to the training image.

[0026] According to a fourth aspect of the present invention, an image resolution adjustment system for improving the image resolution of a digital image is provided. The system includes an upsampling module for performing bicubic upsampling on the digital image to generate a base high-resolution (HR) image. The system also includes a color space processing module for converting the digital image from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. The system also includes a convolution module for converting the LR residual image into a plurality of HR residual sub-images corresponding to the digital image. The system also includes an addition module for generating an HR image corresponding to the digital image using the base HR image and the plurality of HR residual sub-images.

[0027] Any of the above examples can be combined with any one or more of the other examples described above to create new embodiments within the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views.The drawings illustrate generally, by way of example, and not by way of limitation, various embodiments described herein.

[0029] Figure 1 is a block diagram of using a DL architecture (DLA) to train a deep learning (DL) model, as provided by some example embodiments.

[0030] Figure 2 is a diagram of generating a trained DL model using a neural network model trained within DLA, provided by some example embodiments.

[0031] Figure 3 Some example embodiments provide for use Figure 1 Diagram of the DLA system that adjusts image resolution.

[0032] Figure 4 Some example embodiments provide Figure 3 The configuration of the convolutional layers within the DLA used by the system.

[0033] Figure 5 Some example embodiments provide methods for training Figure 1 Figure 1. The generated graph of the training image pairs of DLA.

[0034] Figure 6is a flowchart of generating training image pairs and using these pairs to perform DLA training and image resolution adjustment functions provided by some example embodiments.

[0035] Figure 7 is a flowchart of a method for improving image resolution of a digital image provided by some example embodiments.

[0036] Figure 8 is a block diagram of a representative software architecture provided by some example embodiments that may be used in conjunction with the various device hardware described herein.

[0037] Figure 9 is a block diagram of circuits of devices implementing algorithms and executing methods provided by some example embodiments. DETAILED DESCRIPTION

[0038] It should be understood at the outset that even though the following provides illustrative implementations of one or more embodiments, Figures 1-9 The disclosed systems and methods described can be implemented using any number of techniques, whether currently known or not yet in existence. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0039] The following description is in conjunction with the accompanying drawings, which form a part of the description and show, by way of illustration, specific embodiments in which the present invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the subject matter of the present invention, and it is understood that other embodiments may be utilized and that structural, logical, and electrical changes may be made without departing from the scope of the present invention. Therefore, the following description of the exemplary embodiments is not to be construed in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0040] The present invention relates to video data processing in a computing device. Some aspects relate to changing the image resolution of an image using a neural network. Other aspects relate to real-time video super-resolution.

[0041] As used herein, the terms "low resolution" (or LR) and "high resolution" (or HR) with respect to images are associated with the size of the image (in pixels). For example, if two images depict the same scene, but the height and width (in pixels) of the first image are larger than the height and width of the second image, the first image is referred to as the high-resolution image, and the second image is referred to as the low-resolution image.

[0042] The term “super-resolution” (or SR) as used herein refers to resolution enhancement techniques that increase the number of pixels (e.g., by upscaling) and improve peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) compared to traditional interpolation methods.

[0043] The term “super-resolution” (or UR) as used in this paper includes image resolution enhancement techniques of SR (i.e., increasing the resolution of video frames along with PSNR and SSIM), but also reduces the noise level, removes blocking artifacts typically caused by video compression, and enhances local contrast (these features are not present in SR).

[0044] As used herein, the terms "forward computation" and "backward computation" refer to computations performed in a worker machine in connection with training a neural network model (or another model). The computations performed during forward and backward computations modify weights based on the results of previous iterations (e.g., based on the gradients generated at the end of the previous backward computation). A gradient is a measure of how the output of a worker machine changes when the model weights it is calculating change. The gradient measures how all weights change relative to the change in error. The larger the gradient value, the faster the model learns.

[0045] As used herein, the term "bicubic upsampling" refers to image upsampling using bicubic interpolation. Bicubic interpolation is a technique for interpolating data points on a two-dimensional grid. In image processing, bicubic interpolation considers 16 pixels (a 4×4 pixel matrix) and interpolation is performed using Lagrange polynomials, cubic splines, or cubic convolution algorithms.

[0046] The technology disclosed herein can be used to improve image resolution on mobile devices in real time to achieve super-resolution. The functions associated with UR are accomplished using a simple artificial neural network (ANN) using a neural network model that extracts and propagates residual image information through a limited number of convolutional layers. More specifically, the residual image is obtained by using an ANN (for example, using an LR residual image corresponding to an LR input image), which is the difference between the HR image corresponding to the LR input image and a bicubic upsampled version of the LR input image. Processing the LR residual image using a neural network model reduces the data flow through the convolutional layers, allowing the model to use very few parameters. The terms "neural network" (or NN) and "artificial neural network" (or ANN) are synonyms and are used interchangeably in this article. The terms "convolutional neural network" (or CNN) and "deep neural network" (or DNN) are synonyms and refer to a neural network that includes multiple convolutional layers.

[0047] The technology disclosed herein can also be used to train a neural network model using input-output image pairs created from a training image set. The example images in the training set are downsampled and degraded using simulated noise and block artifacts on the input side to enable the model to perform SR processing, noise reduction (NR) and artifact removal (AR) to achieve optimal UR processing in a mobile device. The local contrast of the HR image on the output side can be enhanced to enhance the details found during the SR processing. In this regard, the neural network model is trained to perform UR processing efficiently with significant visual perception improvements, which can be achieved in real time with acceptable energy consumption for UR processing.

[0048] Existing technologies use image deconvolution algorithms or example-based sparse coding algorithms to perform SR processing. For example, some existing technologies use a three-layer convolution model to simulate the sparse coding process for patch extraction, nonlinear mapping, and reconstruction respectively. However, because the input LR image needs to be upsampled to a high resolution before being fed into the convolution layer, such technologies are inefficient. Therefore, all convolution operations are performed in the HR space, which contains a large number of redundant calculations. Other existing technologies use the LR input image and perform convolution in the LR space until the last step, when the sub-pixels are combined into an HR image using a convolution operation. However, such technologies are also slow when working on mobile devices.

[0049] A new trend in single-image SR processing is to use generative adversarial networks (GANs) to generate missing details in images. However, GANs rely on large network capacity to produce good results and are therefore not suitable for deployment on mobile devices. Other existing SR algorithms designed for video include optical flow estimation between consecutive frames to compensate for motion between frames. Since optical flow estimation is a computationally intensive task that is no easier than processing the tasks associated with using GANs, these video SR algorithms are also slow to work on mobile devices.

[0050] The following are unique features of the technology disclosed in the present invention for improving image resolution and performing real-time UR processing on mobile devices (these features do not exist in prior art image processing technologies), including: (1) using super-resolution processing in mobile device applications, which enhances overall visual perception in addition to the unique task of super-resolution; (2) using a concise neural network model (with reduced number of layers and parameters) that combines the functions of super-resolution, denoising, block artifact removal and local contrast enhancement, and can be deployed on mobile devices with real-time performance (the residual image, rather than the entire SR image, is processed by the convolution layer, saving a lot of processing resources and improving the UR processing speed); (3) using fast pixel shifting operations at the output of the convolution layer to replace the computationally intensive sub-pixel convolution used in traditional technologies; and (4) using a new neural network model training technology to degrade the image quality on the input side while enhancing the image quality on the output side, which facilitates more efficient model training and generation of output video frames with excellent visual perception.

[0051] Figure 1 A framework 100 is shown in a block diagram of some example embodiments for training a deep learning (DL) model using a DL architecture (DLA) to generate a trained DL model 110. In some example embodiments, a machine-learning program (MLP) (including a deep learning program, also collectively referred to as a machine learning algorithm or tool) is used to perform operations associated with correlating data or other artificial intelligence (AI)-based functions.

[0052] like Figure 1 As shown, deep learning model training 108 is performed within DLA 106 based on training data 102 (which may include features). During deep learning model training 108, features from training data 102 may be evaluated to further train the DL model. DL model training 108 produces a trained DL model 110. Trained DL model 110 may include one or more classifiers 112 that may be used to provide an evaluation 116 based on new data 114.

[0053] In some aspects, the training data 102 may include low-resolution (LR) input images 104 and corresponding high-resolution (HR) target output images 118. The LR input images 104 and the HR target output images 118 may be combined, for example, Figure 6The generated and used during DL model training 108 enables the trained DL model 110 to perform the UR related functions described herein. More specifically, the LR input image 104 and the HR target output image 118 are used to train a neural network model (e.g., Figure 2 The neural network model 204 includes Figure 3 A convolutional layer 306 of the convolutional layer 306 is provided to perform a real-time UR function on the LR image including the LR video frame.

[0054] Deep learning is a part of machine learning, a field of study that enables computers to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms (also referred to herein as tools) that can learn from existing data, correlate data, and make predictions about new data. Such machine learning tools work by building a model based on example training data (e.g., training data 102) in order to make data-driven predictions or decisions represented by outputs or evaluations 116. Although example embodiments are presented for some machine learning tools (e.g., deep learning architectures), the principles presented herein may also be applicable to other machine learning tools.

[0055] In some example embodiments, various machine learning tools may be used. For example, logistic regression, Naive-Bayes, random forest (RF), neural network, matrix factorization, and support vector machine (SVM) tools may be used during deep learning model training 108 (e.g., to correlate training data 102).

[0056] Two common types of problems in machine learning are classification and regression. Classification problems aim to classify an item into one of several categorical values (e.g., is this object an apple or an orange?). Regression algorithms aim to quantify certain items (e.g., by providing real-valued values). In some embodiments, DLA 106 can be used to employ machine learning algorithms that use training data 102 to find correlations between identified features that influence the results.

[0057] The machine learning algorithm uses features from the training data 102 to analyze new data 114 to generate estimates 116. These features include individual measurable properties of the observed phenomenon used to train the machine learning model. The concept of a feature is related to the concept of an explanatory variable used in statistical techniques such as linear regression. Selecting informative, discriminative, and independent features is important for the efficient operation of the MLP in pattern recognition, classification, and regression. Features can be of different types, such as numeric features, strings, and graphics. In some aspects, the training data can be of different types, and the features are numeric for use by the computing device.

[0058] In some aspects, the features used during DL model training 108 may include one or more of the following: LR images (e.g., LR input images 104); HR images (e.g., HR target output images 118); sensor data from multiple sensors (e.g., audio sensors, motion sensors, image sensors); brake event data from multiple actuators (e.g., wireless switches or other actuators); external information sources from multiple external sources; timer data associated with sensor state data (e.g., acquisition time sensor data), actuator event data, or external information source data; user communication information; user data; user behavior data, etc.

[0059] The machine learning algorithm uses the training data 102 to find correlations between identification features that influence the outcome of the evaluation 116. In some example embodiments, the training data 102 includes labeled data, which is known data for one or more identification features and one or more outcomes. The DL model training 108 within the DLA 106 trains the DL model using the training data 102 (which may include identification features). The result of the training is a trained DL model 110. When the DL model 110 is used for evaluation, new data 114 is provided as input to the trained DL model 110, which generates an evaluation 116 as output. For example, the DLA 106 can be deployed on a mobile device, and the new data 114 can include LR images (e.g., frames from an LR video, such as a real-time LR video feed). The DLA 106 performs UR functions (e.g., increasing image resolution while reducing noise, removing blocking artifacts, and improving image contrast) on the LR images to generate an HR output image in real time.

[0060] Figure 2 FIG. 2 is a diagram illustrating a framework 200 for generating a trained DL model 206 using a neural network model 204 trained within a DLA 106, as provided by some example embodiments. Figure 2, source data 202 can be analyzed by a neural network model 204 (or another type of machine learning algorithm or technique) to generate a trained DL model 206 (which can be the same as the trained DL model 110). Source data 202 can include a training data set (e.g., 102), including data identified by one or more features. As used herein, the terms "neural network" and "neural network model" are interchangeable.

[0061] Machine learning techniques train models to accurately predict data that is input into the model (e.g., what the user said in a given conversation sentence; whether a noun is a person, place, or thing; what the weather will be like tomorrow). During the learning phase, the model is developed based on the input training data set to optimize the model so that it correctly predicts the target output for the given input. Typically, the learning phase can be supervised, semi-supervised, or unsupervised; indicating the level of decline in providing the "correct" output corresponding to the training input. In the supervised learning phase, all target outputs are provided to the model, and the model is guided to develop general rules or algorithms that map inputs to outputs. In contrast, in the unsupervised learning phase, the desired output is not provided for the input so that the model can develop its own rules to discover relationships within the training data set. In the semi-supervised learning phase, an incompletely labeled training set is provided, where some outputs of the training data set are known and some are unknown.

[0062] The model can be run against a training dataset for several cycles, where the training dataset is repeatedly input into the model to refine its results (i.e., the entire dataset is processed in one cycle). During iterations, the model (e.g., a neural network model or another machine learning model) is run against small batches (or portions) of the entire dataset. In the supervised learning phase, the model is developed to predict the target output for a given set of inputs (e.g., source data 202) and evaluated over several cycles to more reliably provide an output that is specified as corresponding to the given input for the maximum number of inputs of the training dataset. In another example, for the unsupervised learning phase, a model is developed to cluster the dataset into n groups, and the consistency of the model in placing the given input into the given group and the reliability of the model in generating the n desired clusters in each cycle are evaluated over several cycles.

[0063] After running a cycle, the model is evaluated and the values of the variables of these models (e.g., weights, biases, or other parameters) are adjusted to try to better refine the model through iteration. As used herein, the term "weights" refers to the parameters used by the machine learning model. During the backward calculation, the model can output gradients, which can be used to update the weights associated with the forward calculation.

[0064] In various aspects, the assessment is biased towards false negatives, biased towards false positives, or uniformly biased towards the overall accuracy of the model. Depending on the machine learning technique used, the values can be adjusted in a variety of ways. For example, in a genetic algorithm or evolutionary algorithm, the values of the model that most successfully predicts the desired output are used to develop the values used by the model in subsequent cycles, and these values may include random changes / mutations to provide additional data points. Those of ordinary skill in the art are familiar with several other machine learning algorithms that can be applied to the present invention, including linear regression, random forests, decision tree learning, neural networks, deep neural networks, etc.

[0065] Each model develops rules or algorithms over several cycles by changing the values of one or more variables affecting the input to more closely map to the desired outcome, but since the training dataset can vary and is preferably very large, perfect accuracy and precision may not be achieved. Therefore, some of the cycles that make up the learning phase can be set to a given number of trials or a fixed time / computational budget, or can be terminated before reaching that number / budget when the accuracy of a given model is sufficiently high or low or reaches a plateau in accuracy. For example, if the training phase is designed to run for n cycles and generate a model with at least 95% accuracy, and such a model is generated before the nth cycle, the learning phase can be terminated early and the generated model that meets the final target accuracy threshold can be used. Similarly, if the accuracy of a given model is insufficient to meet a random chance threshold (e.g., the model is only 55% accurate in determining a true / false output for a given input), the learning phase for that model can be terminated early, but training of other models in the learning phase can continue. Similarly, when a given model consistently provides similar accuracy or fluctuations in results over multiple epochs (having reached a plateau in performance), the learning phase for a given model may terminate before the epoch number / computational budget is reached.

[0066] After the learning phase is complete, the model is finalized. In some example embodiments, the finalized model is evaluated based on testing criteria. In a first example, a test dataset consisting of known target outputs for the input is fed into the finalized model to determine the model's accuracy when processing untrained data. In a second example, false positive or false negative rates can be used to evaluate the finalized model. In a third example, the demarcation between data clusters in each model is used to select the model that produces the clearest boundaries for its data clusters.

[0067] In some example embodiments, the DL model 206 is trained by a neural network model 204 (e.g., a deep learning network, a deep convolutional network, or a recursive neural network) that includes a series of "neurons" (e.g., long short term memory (LSTM) nodes) arranged into a network. A neuron is an architectural element for data processing and artificial intelligence, particularly machine learning, that includes a memory that can determine when to "remember" and when to "forget" the values stored in that memory based on the weights of the inputs provided to a given neuron. Each neuron as used herein is used to receive a predefined number of inputs from other neurons in the network in order to provide relation and sub-relation outputs for the content of the frame being analyzed. Individual neurons can be linked together or organized into a tree structure in various configurations of the neural network to provide interaction and relational learning modeling to determine how each frame in a conversation sentence is related to each other.

[0068] For example, an LSTM as a neuron includes several gates for processing an input vector (e.g., a phoneme from a conversation sentence), a memory cell, and an output vector (e.g., a context representation). The input gate and output gate control the flow of information into and out of the memory cell, respectively, while the forget gate optionally deletes information from the memory cell based on the input of earlier linked units in the neural network. The weights and bias vectors of the various gates are adjusted during the training phase, and once the training phase is complete, these weights and biases are finalized for normal operation. Those skilled in the art will understand that neurons and neural networks can be constructed programmatically (e.g., through software instructions) or by connecting each neuron to form a dedicated hardware of the neural network.

[0069] Neural networks use features to analyze data to generate estimates (e.g., identify speech units). A feature is a single, measurable property of the observed phenomenon. The concept of a feature is related to the concept of an explanatory variable used in statistical techniques such as linear regression. Furthermore, deep features represent the outputs of nodes in the hidden layers of a deep neural network.

[0070] Neural networks (e.g., neural network model 204), sometimes called artificial neural networks or neural network models, are computing systems based on biological neural networks in the brains of animals. These systems gradually improve in performance (i.e., learn) to perform tasks, typically without the need for task-specific programming. For example, in image recognition, a neural network can be taught to recognize images containing objects by analyzing example images that have been labeled with the names of the objects and learning the objects and names, and the neural network can use the analysis results to recognize objects in unlabeled images. A neural network is based on a collection of connected units called neurons, where each connection between neurons (called a synapse) can transmit a unidirectional signal whose activation strength varies with the strength of the connection. A receiving neuron can activate and propagate a signal to downstream neurons connected to it, typically based on whether the combined input signal from potentially many transmitting neurons has sufficient strength, where strength is a parameter.

[0071] A DNN, also known as a CNN, is a type of neural network constructed from multiple stacked convolutional layers. These layers consist of nodes, where computation occurs, loosely arranged to form a pattern similar to neurons in the human brain, which fire when exposed to sufficient stimulation. A node combines data input with a set of coefficients, or weights, that can amplify or suppress that input, thereby assigning importance to the input for the task the algorithm is trying to learn. These input-weight products are summed and passed through the node's activation function to determine whether and to what extent the signal should pass further through the network to influence the final outcome. DNNs use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as input. High-level features are derived from lower-level features to form a hierarchical representation. Layers following the input layer may be convolutional layers, which produce feature maps, which are filtered versions of the input and used by the next convolutional layer.

[0072] In the training of DNN architectures, regression can include minimizing a cost function, where regression is constructed as a set of statistical procedures for estimating the relationship between variables. The cost function can be implemented as a function that returns a numerical value representing the performance of the neural network in mapping training examples to correct the output. During training, if the cost function value is not within a predetermined range, backpropagation is used based on known training images, where backpropagation is a common method for training artificial neural networks and is used together with optimization methods such as stochastic gradient descent (SGD).

[0073] Backpropagation can be used to both propagate and update weights. When an input is presented to a neural network, it is propagated forward through the network layer by layer until it reaches the output layer. The output of the neural network is then compared to the desired target output using a cost function, and an error value is calculated for each node in the output layer. The error values are propagated backward from the output until each node has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values to calculate the gradient of the cost function with respect to the weights in the neural network. The calculated gradient is fed into the chosen optimization method to update the weights in an attempt to minimize the cost function.

[0074] Even though the training architecture 106 is referred to as a deep learning architecture using a neural network model (and the trained models are referred to as trained deep learning models, such as the trained DL models 110 and 206), the present invention is not limited in this regard, and other types of machine learning training architectures may also be used for model training using the techniques disclosed herein.

[0075] Figure 3 Some example embodiments provide for use Figure 1 FIG300 is a diagram of a system for adjusting image resolution using a DLA. Figure 3 , the system 300 includes a convolution layer 306, a residual generation module 322, a bicubic upsampling module 316, a pixel shift module 318 and an adder 320. The convolution layer 306 can be configured as a neural network model, such as Figure 2 Neural network model 204.

[0076] The residual generation module 322 may include suitable circuitry, logic, interfaces, or code for converting the input LR image 302 from a red-green-blue (RGB) color space into a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (or YCbCr) color space, and obtaining a Y channel (luminance) LR residual image 304 (i.e., a grayscale image indicating luminance) corresponding to the input LR image 302.

[0077] The bicubic upsampling module 316 may comprise suitable circuitry, logic, interfaces or code that may be operable to perform bicubic upsampling on the input LR image 302 to generate the base HR image 312 .

[0078] The convolutional layer 306 may be part of a neural network model (e.g., the neural network model 204) for generating a plurality of HR residual sub-images 308 corresponding to the input LR image 302 based on the LR residual image (e.g., a grayscale version of the LR input image (e.g., the LR residual image 304)). Figure 4 Further details of the convolutional layer 306 are described.

[0079] The pixel shifting module 318 may comprise suitable circuitry, logic, interfaces or code that may be operable to perform pixel shifting on the plurality of HR residual sub-images 308 to generate the HR residual image 310 .

[0080] In operation, a low-resolution (LR) input image 302 is processed (e.g., by a bicubic upsampling module 316 and a residual generation module 322) to generate a base HR image 312 and an LR residual image 304 corresponding to the input LR image 302. The convolutional layer 306 uses multiple layers to convolve the LR residual image 304 and generate a plurality of HR residual sub-images 308 corresponding to the input LR image 302. In some aspects, the plurality of HR residual sub-images 308 include four sub-images. The pixel shift module 318 performs pixel shifting on the plurality of HR residual sub-images 308 to generate an HR residual image 310. The adder 320 may comprise suitable circuitry, logic, interfaces, or code for adding the base HR image 312 to the HR residual image 310 to generate the HR image 314 as an output image corresponding to the input LR image 302.

[0081] By splitting the generation of the HR image 314 into generating a base HR image 312 and an HR residual image 310, the amount of data flowing through the convolutional layer 306 is reduced, thereby improving the effective capacity of the model and its inference efficiency in reconstructing an HR image based on an input LR image.

[0082] In some aspects, one or more functions performed by the bicubic upsampling module 316 , the residual generation module 322 , the pixel shift module 318 , and the adder 320 may be performed by the convolutional layer 306 .

[0083] Figure 4 Some example embodiments provide Figure 3 The system 300 uses the configuration of the convolutional layer 306 within the DLA. Figure 4 , the convolutional layer 306 includes four layers, namely convolutional layers 404, 406, 408 and 410, with a layer depth (or number of channels) 420, as shown Figure 4 As shown in FIG. 1 , the first convolutional layer 404 is used to generate 8 measurement channels (i.e., the depth of the layer is 8), the second convolutional layer 406 is used to generate 6 measurement channels (i.e., the depth of the layer is 6), and the third and fourth convolutional layers (408 and 410, respectively) are each used to generate 4 measurement channels (i.e., the depth of the layer is 4). Convolutional layers 404-408 are used to generate their channel measurements using convolution kernels 414, 416, and 418 of size 3×3 pixels, respectively.

[0084] The convolution layer 306 may further include a single-channel input layer 402 corresponding to an image received as input to the convolution layers 404-410. For example, the input layer 402 may represent the LR residual image 304 transmitted as input to the convolution layers 404-410. The input layer 402 may include a kernel 412 (e.g., a 5×5 pixel kernel) for transmitting image data as input to the first convolution layer 404.

[0085] In operation, each of the convolutional layers 404-410 convolves the input received from the previous layer using multiple convolution kernels. For example, after the first convolutional layer 404 receives input data from the input layer 402 through the kernel 412, the first convolutional layer 404 generates 8 measurement channels using the input of the input layer 402 and eight different convolution kernels of size 3×3 pixels (such as the convolution kernel 414). The measurement of each channel is passed to the subsequent layer for additional convolution. The fourth convolutional layer 410 outputs multiple HR residual sub-images 308 (e.g., each of the four channels of the layer 410 outputs one HR residual sub-image for a total of four HR residual sub-images 308).

[0086] although Figure 4 Convolutional layers 306 are shown including four separate convolutional layers 404-410 with indicated layer depths 420, but the invention is not limited in this respect and different configurations of convolutional layers 306 may also be used within system 300 to perform UR-related functions.

[0087] Figure 5 Some example embodiments provide methods for training Figure 1 The DLA training image pairs are generated using the framework shown in FIG500. Figure 5 , each training image pair includes a convolution layer input image 532 and a set of convolution layer target output images 536. The convolution layer input image 532 and the convolution layer target output image 536 are both input to the convolution layer 306 during the training phase, and the convolution layer target output image 536 does not include the actual output during the training phase. It should be understood that during the actual convolution operation, the image to be processed is input to the convolution layer 306 at 532, and the convolution layer 306 will output one or more processed UR images at 536.

[0088] To generate the convolutional layer input image 532, a low-pass filter (LPF) 522 is applied to the example training image I 502 to generate a filtered image 504. The filtered image 504 is then downsampled (e.g., using a downsampling (DS) module 524) to generate a downsampled LR image I LR506. In some aspects, the filtered image 504 may be downsampled by a factor of 0.5, as represented by the following equation: LR =I*N(0,σ blur )↓ 0.5 , where * represents the convolution operation, N(0,σ blur ) means the mean is 0 and the standard deviation is σ blur Gaussian function.

[0089] In some aspects, the LPF 522 is used to remove high frequency signals, thereby avoiding downsampling artifacts. In some aspects, the downsampling of the DS module 524 is performed by bicubic interpolation. The downsampled (e.g., half-sized) LR image 506 is then degraded by noise and blocking artifacts before being fed to the convolutional layer. More specifically, the noise addition module 526 is used to introduce noise into the downsampled LR image 506. In some aspects, two types of noise are simulated, namely (1) photon noise and (2) Gaussian noise.

[0090] Photon noise is used to simulate the discrete characteristics of light and is simulated using a Poisson stochastic process as follows: pho ~P(I LR ·photons) / photons, where P(λ) represents a Poisson distribution with mean λ and photons represents the number of photons that produce the brightest color that can be represented in the image.

[0091] Gaussian noise is used to simulate the random noise of the image sensor. It is generated by a random image I generated by a Gaussian random process. gauss It is calculated by adding it to the input image as follows: noisy =I pho +I gauss , I gauss ~N(0,σ n ), where σ n Indicates the target noise level.

[0092] Video compression and decompression often result in blocking artifacts in video frames. Artifacts are directly simulated by video compression and decompression. Training a neural network model (e.g., 204) with convolutional layers 306 that degrade image quality on the input side will give the neural network model noise reduction and artifact removal capabilities.

[0093] Noise addition module 526 introduces noise into the downsampled LR image 506 to generate a noisy LR image 508. Noisy LR image 508 is further degraded by artifact addition module 528, which introduces artifacts and generates an LR image 510 corresponding to training image 502. LR image 510 is characterized by degraded image quality and is configured as a convolutional layer input image 532 that enters convolutional layer 306 during the training phase. Convolutional layer input image 532 is used to train convolutional layer 306 to generate a desired target output. Here, during the training phase of the convolutional layer, the desired output of convolutional layer 306 is input as a convolutional layer target output image 536.

[0094] To generate a set of target output images 536 for the convolutional layers of the convolutional network, the local contrast enhancement (LCE) module 520 enhances the training images I 502 to generate contrast-enhanced images 512. This process helps teach the convolutional layers 306 to produce images with high local contrast, which improves the details recovered by super-resolution. In some aspects, the LCE module 520 can perform local contrast enhancement by applying an unbalanced unsharp mask as follows: lc =I+min(I–I*N(0,σ um ),δ max ), where the difference between the training image and the Gaussian blurred version of the training image is upper bounded by δ max Truncated in the positive part (truncation helps reduce halo artifacts that appear due to the human eye's sensitivity to bright edges). The subtraction module 540 then calculates the HR residual image 516 by subtracting the upsampled LR image 538 of the downsampled LR image 506 from the contrast enhanced image 512. The upsampled LR image 538 is generated by upsampling the downsampled LR image 506 by the upsampling module 530. A pixel partitioning operation 534 is applied to the HR residual image 516 to generate a plurality of HR residual sub-images 518 from the HR residual image 516 (for example, by partitioning a 4×4 pixel block into four sub-images). The plurality of HR residual sub-images 518 are configured as the convolution layer target output image 536, which is input into the convolution layer 306 during the training phase.

[0095] In some aspects, multiple training pairs are generated before training the convolutional layer 306 and saved in a data file (e.g., using the TF record format). During neural network model training (e.g., 108), the training code can extract the training data and use the data to adjust the model parameters used to perform UR processing (e.g., the model parameters of the convolutional layer 306). This speeds up model training by avoiding complex image processing in real time and makes it possible to repeatedly train the model using different parameters associated with the training data.

[0096] By using the neural network model training techniques described herein (e.g., degrading image quality on the input side and enhancing image quality on the output side), the convolution layer 306 can generate images or video frames with excellent visual perception during UR processing of LR images or video frames. In addition, using a concise neural network model (e.g., a convolution layer 306 having a limited number of layers, such as four layers), which combines the functions of super-resolution, noise reduction, block artifact removal, and local contrast enhancement, the concise neural network model can be deployed on mobile devices (or other types of limited resource devices, such as smart TVs, tablets, laptops, and other computing devices) to perform real-time UR processing.

[0097] In operational use, the actual image or video frame to be processed will comprise the input 532 of the convolutional layer 306. As a result of the training, the convolutional layer 306 will process the image or video frame to generate a UR image or video frame at the output 536. The output UR image or video frame may then be displayed, transmitted, stored, or otherwise used.

[0098] Figure 6 6 is a flow chart of a method 600 for generating training image pairs and using these pairs for DLA training and performing image resolution adjustment functions according to some example embodiments. The method 600 includes operations 602-618. By way of example and not limitation, the method 600 is described as being performed by the super-resolution management module 860. The super-resolution management module 860 can be used in Figure 9 The device 900 shown is executed in a mobile device.

[0099] refer to Figure 6, the generation of the training image pair may begin when a training image (e.g., 502) is received in operation 602. In operation 604, the training image is filtered using a low-pass filter (e.g., 522) to generate a filtered image (e.g., 504). In operation 606, the filtered image is downsampled (e.g., by a downsampling module 524) to generate a downsampled LR image (e.g., 506). In operation 608, the image quality of the downsampled LR image is degraded by adding noise (e.g., by a noise addition module 526) and artifacts (e.g., by an artifact addition module 528) to generate a low-resolution image (e.g., LR image 510) corresponding to the training image and having degraded image quality. In operation 610, the LR image corresponding to the training image is configured as an input to the neural network model.

[0100] To generate an output image within a training pair, in operation 612, an unbalanced unsharp mask is applied to the training image to generate a contrast-enhanced image (e.g., the LCE module 520 generates the contrast-enhanced image 512). In operation 614, an upsampled version of the downsampled LR image is subtracted from the contrast-enhanced image to generate an HR residual image corresponding to the training image (e.g., the HR residual image 516 is generated by the subtraction module 540 using the upsampled LR image 538 and the contrast-enhanced image 512). In operation 616, the HR residual image (corresponding to the training image) is partitioned to generate a plurality of HR residual sub-images corresponding to the training image (e.g., the plurality of HR residual sub-images 518 are generated by the pixel partitioning operation 534 performed on the HR residual image 516). In operation 618, the plurality of HR residual sub-images corresponding to the training image are configured as outputs of the neural network model (e.g., the plurality of HR residual sub-images 518 are configured as a set of convolutional layer target output images 536 for training the convolutional layer 306).

[0101] Figure 7 700 is a flow chart of a method 700 for improving the image resolution of a digital image provided by some example embodiments. The method 700 includes operations 702, 704, 706, and 708. By way of example and not limitation, the method 700 is described as being performed by a super-resolution management module 860. The super-resolution management module 860 may be used in Figure 9 The device 900 shown is executed in a mobile device.

[0102] refer to Figure 7 In operation 702, the digital image is bicubicly upsampled to generate a base high-resolution (HR) image. Figure 3, the bicubic upsampling module generates a base HR image using the input LR image. In operation 704, the digital image is converted from the red-green-blue (RGB) color space to the luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image. In some aspects, the RGB to YCrCb conversion can be performed by applying multiple convolutional layers of a neural network to the digital image (or by using a residual generation module) to generate the LR residual image.

[0103] In operation 706, the LR residual image is converted (e.g., by using multiple convolutional layers) into multiple HR residual sub-images corresponding to the input LR image. For example, the convolutional layer generates multiple HR residual sub-images corresponding to the input LR image using the LR residual image. In operation 708, an HR image corresponding to the input LR image is generated using the base HR image and the multiple HR residual sub-images. For example, the HR residual image is generated by pixel-shifting the multiple HR residual sub-images. The HR image is generated using the base HR image and the HR residual image.

[0104] Figure 8 is a block diagram of a representative software architecture 800 provided by some example embodiments, which may be used in conjunction with the various device hardware described herein. Figure 8 This is merely a non-limiting example of a software architecture 802, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Figure 9 The system is executed in hardware such as the device 900, which includes a processor 905, a memory 910, storage 915 and / or 920, and I / O interfaces 925 and 930.

[0105] A representative hardware layer 804 is shown, which may represent Figure 9 The representative hardware layer 804 includes one or more processing units 806 with associated executable instructions 808. The executable instructions 808 represent executable instructions of the software architecture 802, including the implementation of Figures 1-9 The hardware layer 804 also includes a memory or storage module 810, which also has executable instructions 808. The hardware layer 804 may also include other hardware 812, which represents any other hardware of the hardware layer 804, such as other hardware shown as part of the device 900.

[0106] exist Figure 8In the example architecture of FIG, software architecture 802 can be conceptualized as a stack of layers, where each layer has specific functionality. For example, software architecture 802 may include layers such as operating system 814, library 816, framework / middleware 818, application 820, and presentation layer 844. In operation, application 820 or other components within each layer can call application programming interface (API) call 824 through the software stack and receive responses, return values, etc., shown as messages 826 in response to API call 824. Figure 8 The layers shown are representative in nature, and not all software architectures 802 have all layers. For example, some mobile or specialized operating systems may not provide framework / middleware 818, while other operating systems may provide such layers. Other software architectures may include additional or different layers.

[0107] The operating system 814 can manage hardware resources and provide public services. The operating system 814 may include a kernel 828, services 830, and drivers 832. The kernel 828 may serve as an abstraction layer between the hardware and other software layers. For example, the kernel 828 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 830 may provide other public services to other software layers. The drivers 832 may be responsible for controlling or connecting to the underlying hardware. For example, depending on the hardware configuration, the drivers 832 may include a display driver, a camera driver, drives, flash drives, serial communication drivers (e.g., universal serial bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0108] The library 816 may provide a common infrastructure that can be used by applications 820 or other components or layers. The function of the library 816 is generally to enable other software modules to perform tasks more easily than by directly connecting to the underlying operating system 814 functions (e.g., kernel 828, services 830, or drivers 832). The library 816 may include a system library 834 (e.g., a C standard library), which may provide functions such as memory allocation, string manipulation, and mathematical functions. In addition, the library 816 may include an API library 836, such as a media library (e.g., a library that supports the rendering and manipulation of various media formats (e.g., MPEG4, H.264, MP3, AAC, AMR, JPG, PNG)), a graphics library (e.g., an OpenGL framework that can be used to render 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a web library (e.g., WebKit that can provide web browsing functions), etc. The library 816 may also include a variety of other libraries 838 to provide many other APIs to the applications 820 and other software components / modules.

[0109] Framework / middleware 818 (sometimes also referred to as middleware) can provide a high-level common infrastructure that can be used by applications 820 or other software components / modules. For example, framework / middleware 818 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. Framework / middleware 818 can provide a wide range of other APIs that can be used by applications 820 or other software components / modules, some of which may be specific to a particular operating system 814 or platform.

[0110] The applications 820 include built-in applications 840, third-party applications 842, and an ultra-resolution management module (URMM) 860. In some aspects, the URMM 860 may include suitable circuitry, logic, interfaces, or code, and may be configured to perform the combined Figure 1-Figure 7 Describes one or more UR-related functions.

[0111] Examples of representative built-in applications 840 may include, but are not limited to, a contacts application, a browser application, a reader application, a location application, a media application, a messaging application, or a game application. Third-party applications 842 may include any of the built-in applications 840 as well as a wide variety of other applications. In a specific example, third-party applications 842 (e.g., applications developed by entities other than the vendor of a particular platform using Android TM or iOS TMApplications developed with a software development kit (SDK) can be developed on platforms such as iOS TM 、Android TM 、 Mobile software running on a mobile operating system such as a phone or other mobile operating system. In this example, third-party applications 842 can call API calls 824 provided by a mobile operating system (such as operating system 814) to facilitate the functionality described herein.

[0112] Applications 820 can create user interfaces to interact with system users by utilizing built-in operating system functionality (e.g., kernel 828, services 830, and drivers 832), libraries (e.g., system libraries 834, API libraries 836, and other libraries 838), and framework / middleware 818. Alternatively or additionally, in some systems, user interaction can occur through a presentation layer (e.g., presentation layer 844). In these systems, application / module "logic" can be separated from aspects of the application / module that interact with the user.

[0113] Some software architectures use virtual machines. Figure 8 In the example of FIG, a virtual machine is shown by virtual machine 848. A virtual machine creates a software environment in which applications / modules can be run as if they were on a hardware machine (e.g., Figure 9 The virtual machine 848 is hosted by a host operating system (e.g., operating system 814) and typically (though not always) includes a virtual machine monitor 846. The virtual machine monitor 846 manages the operation of the virtual machine 848 and its connection to the host operating system (i.e., operating system 814). The software architecture 802 is executed within the virtual machine 848, such as the operating system 850, libraries 852, framework / middleware 854, applications 856, or presentation layer 858. These software architecture layers executed within the virtual machine 848 can be the same as or different from the corresponding layers described above.

[0114] Figure 9 is a block diagram of circuitry for devices implementing algorithms and methods provided in some example embodiments. Not all components need be used in various embodiments. For example, a client, server, and cloud-based network device may each utilize a different set of components or, in the case of a server, a larger storage device.

[0115] An example computing device in the form of a computer 900 (also referred to as computing device 900, computer system 900, or computer 900) may include a processor 905, memory 910, removable storage 915, non-removable storage 920, input interface 925, output interface 930, and communication interface 935, all connected via a bus 940. Although the example computing device is shown and described as a computer 900, the computing device may take different forms in different embodiments.

[0116] Memory 910 may include volatile memory 945 and non-volatile memory 950 and may store programs 955. Computing device 900 may include or have access to a computing environment that includes various computer-readable media, such as volatile memory 945, non-volatile memory 950, removable storage 915, and non-removable storage 920. Computer memory includes random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0117] Computer-readable instructions stored in a computer-readable medium (e.g., program 955 stored in memory 910) can be executed by the processor 905 of the computing device 900. Hard drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media (such as storage devices). The terms "computer-readable medium" and "storage device" do not include carrier waves that are considered too ephemeral. "Computer-readable non-transitory medium" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media. It should be understood that the software can be installed in the computer and sold with the computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through physical media or distribution systems, including, for example, obtaining the software from a server owned by the software creator or from a server that the software creator does not own but uses. For example, the software can be stored on a server for distribution over the Internet. As used herein, the terms "computer-readable medium" and "machine-readable medium" are interchangeable.

[0118] The program 955 can utilize the modules described herein (such as URMM 960, which can be used with Figure 8 The consumer preference structure of URMM 860 is the same as that of URMM 860.

[0119] Any one or more modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any suitable combination thereof). In addition, any two or more of these modules may be combined into a single module, and the functionality of a single module described herein may be subdivided between multiple modules. Furthermore, according to various example embodiments, modules described herein as being implemented within a single machine, database, or device may be distributed across multiple machines, databases, or devices.

[0120] In some aspects, the URMM 960 and one or more other modules that are part of the program 955 can be integrated into a single module that performs the corresponding functions of the integrated module.

[0121] Although several embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve the desired results. Additional steps may be provided or deleted from the described flows, and additional components may be added to or removed from the described systems. Other embodiments are within the scope of the following claims.

[0122] It should also be understood that software comprising one or more computer-executable instructions that facilitate the processes and operations described above in connection with any or all of the steps of the present invention can be installed in and sold with one or more computing devices consistent with the present invention. Alternatively, the software can be acquired and loaded onto one or more computing devices, including acquiring the software via physical media or a distribution system, including, for example, from a server owned by the software creator or from a server not owned by the software creator but used by the software creator. For example, the software can be stored on a server for distribution over the Internet.

[0123] In addition, it will be understood by those skilled in the art that the present invention is not limited in its application to the construction and arrangement details of the components described in the specification or shown in the drawings. The embodiments herein are applicable to other embodiments and can be practiced or executed in various ways. In addition, it will be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of "including", "comprising" or "having" and their variations in this article is intended to cover the items listed thereafter and their equivalents as well as other items. Unless otherwise limited, the terms "connect", "couple" and "install" and their variations are widely used in this article and cover direct and indirect connections, couplings and installations. In addition, the terms "connect" and "couple" and their variations are not limited to physical or mechanical connections or couplings. In addition, terms such as "upper", "lower", "bottom" and "top" are relative and are used to help illustrate, but are not restrictive.

[0124] The components of the illustrative apparatus, systems, and methods used in accordance with the described embodiments may be implemented at least in part in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. For example, the components may be implemented as a computer program product (e.g., a computer program, program code, or computer instructions) tangibly embodied in an information carrier or in a machine-readable storage device for execution by a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers), or for controlling the operation of a data processing apparatus.

[0125] A computer program can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can be deployed to be executed in one computer or in multiple computers at one site, or it can be distributed at multiple sites and interconnected via a communication network. In addition, the functional programs, codes, and code segments for implementing the technology described herein are easily understood by programmers in the field of the technology described herein to be within the scope of the claims. The method steps associated with the illustrative embodiments can be performed by one or more programmable processors, thereby executing computer programs, codes, or instructions to perform functions (e.g., operate on input data or generate output). For example, the method steps can also be performed by a dedicated logic circuit (e.g., a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)), and the device for performing the method can be implemented as the dedicated logic circuit.

[0126] The various illustrative logical blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general-purpose processor may be a microprocessor, but may alternatively be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other similar configuration.

[0127] For example, processors suitable for executing computer programs include general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, a processor receives instructions and data from read-only memory or random access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer also includes one or more mass storage devices for storing data (e.g., magnetic disks, magneto-optical disks, or optical disks), or is operably coupled to one or more mass storage devices for storing data to receive data from the mass storage devices and / or transfer data to the mass storage devices. Information carriers suitable for embodying computer program instructions and data include various forms of non-volatile memory, for example, semiconductor memory devices, such as electrically programmable read-only memory or electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory devices, data storage disks (e.g., magnetic disks, internal hard disks, or removable disks, magneto-optical disks, CD-ROMs, and DVD-ROMs). The processor and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0128] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols, and chips referenced in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or optical particles, or any combination thereof.

[0129] As used herein, "machine-readable media" (or "computer-readable media") includes devices capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., erasable programmable read-only memory (EEPROM)), or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store processor instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media that can store instructions executed by one or more processors 905, which, when executed by one or more processors 905, perform any one or more of the methods described herein. Accordingly, "machine-readable media" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" as used herein does not include the signal itself.

[0130] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems, and methods described and illustrated as discrete or separate in the various embodiments may be combined or integrated with other systems, modules, techniques, or methods. Other items shown or described as coupled to each other, or directly coupled to each other, or communicating with each other may be indirectly coupled or communicated with each other electrically, mechanically, or otherwise via some interface, device, or intermediate component. Other examples of variations, substitutions, and modifications may be determined by those skilled in the art and may be exemplified without departing from the scope of the present invention.

[0131] Although the present invention has been described with reference to specific features and embodiments of the present invention, it is obvious that various modifications and combinations of the present invention can be formulated without departing from the present invention. For example, other components can be added to the described system or removed from the described system. Therefore, the description and the drawings are only regarded as an illustration of the present invention as defined by the appended claims and any and all modifications, variants, combinations or equivalents falling within the scope of the present invention are considered. Other aspects may be within the scope of the appended claims. Finally, unless otherwise specifically stated, the conjunction "or" used herein refers to a non-exclusive "or".

Claims

1. A computer-implemented method for improving the image resolution of a digital image, characterized in that: The method comprises: performing bicubic upsampling on the digital image to generate a base high-resolution (HR) image; Converting the digital image from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image; Converting the LR residual image into a plurality of HR residual sub-images corresponding to the digital image using a plurality of convolutional layers of a neural network model; Performing pixel shifting on the multiple HR residual sub-images to generate an HR residual image; The base HR image and the HR residual image are combined to generate an HR image corresponding to the digital image.

2. The computer-implemented method of claim 1, wherein: The neural network model includes an input layer, The multiple convolutional layers include four convolutional layers.

3. The computer-implemented method of claim 2, wherein: The input layer is used to receive the digital image, and the output layers of the four convolutional layers are used to output the multiple HR residual sub-images.

4. The computer-implemented method of claim 2, wherein: A first layer of the plurality of convolutional layers is configured with a 3×3 pixel kernel and 8 channels; A second layer of the plurality of convolutional layers is configured with a 3×3 pixel kernel and 6 channels; A third layer of the plurality of convolutional layers is configured with a 3×3 pixel kernel and 4 channels; A fourth layer of the plurality of convolutional layers is configured with 4 channels.

5. The computer-implemented method according to any one of claims 1 to 4, wherein: Also includes: The neural network model is trained using a plurality of training image pairs, each of the plurality of training image pairs comprising: an LR image corresponding to the training image, the LR image having degraded image quality and configured as an input to the neural network model; A plurality of HR residual sub-images corresponding to the training images and configured as target outputs of the neural network model.

6. The computer-implemented method of claim 5, wherein: Training the neural network model further includes: filtering the training image using a low-pass filter to generate a filtered image; Downsampling the filtered image to generate a downsampled LR image; The image quality of the downsampled LR image is degraded by adding noise and artifacts to generate the LR image corresponding to the training image.

7. The computer-implemented method of claim 6, wherein: Training the neural network model further includes: applying an unbalanced unsharp mask to the training image to generate a contrast enhanced image; An upsampled version of the downsampled LR image is subtracted from the contrast-enhanced image to generate a HR residual image corresponding to the training image.

8. The computer-implemented method of claim 7, wherein: Training the neural network model further includes: The HR residual image corresponding to the training image is divided to generate the plurality of HR residual sub-images corresponding to the training image.

9. A computer system for improving the image resolution of a digital image, characterized in that include: a memory for storing instructions; one or more processors in communication with the memory, wherein the one or more processors execute the instructions to: Perform bicubic upsampling on digital images to generate base high-resolution (HR) images; The digital image is converted from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution ( LR) residual image; Converting the LR residual image into a plurality of HR residual sub-images corresponding to the digital image using a plurality of convolutional layers of a neural network model; Performing pixel shifting on the multiple HR residual sub-images to generate an HR residual image; The base HR image and the HR residual image are combined to generate an HR image corresponding to the digital image.

10. The system according to claim 9, characterized in that: The neural network model includes an input layer, and the multiple convolutional layers include four convolutional layers; The input layer is used to receive the digital image; The output layers of the four convolutional layers are used to output the multiple HR residual sub-images.

11. The system according to claim 10, characterized in that: A first layer of the plurality of convolutional layers is configured with a 3×3 pixel kernel and 8 channels; A second layer of the plurality of convolutional layers is configured with a 3×3 pixel kernel and 6 channels; A third layer of the plurality of convolutional layers is configured with a 3×3 pixel kernel and 4 channels; A fourth layer of the plurality of convolutional layers is configured with 4 channels.

12. The system according to any one of claims 9 to 11, characterized in that The one or more processors execute the instructions to: The neural network model is trained using a plurality of training image pairs, each of the plurality of training image pairs comprising: an LR image corresponding to the training image, the LR image having degraded image quality and configured as an input to the neural network model; A plurality of HR residual sub-images corresponding to the training images and configured as target outputs of the neural network model.

13. The system according to claim 12, wherein: To train the neural network model, the one or more processors execute the instructions to: filtering the training image using a low-pass filter to generate a filtered image; Downsampling the filtered image to generate a downsampled LR image; The image quality of the downsampled LR image is degraded by adding noise and artifacts to generate the LR image corresponding to the training image.

14. The system according to claim 13, wherein: To train the neural network model, the one or more processors execute the instructions to: applying an unbalanced unsharp mask to the training image to generate a contrast enhanced image; An upsampled version of the downsampled LR image is subtracted from the contrast-enhanced image to generate a HR residual image corresponding to the training image.

15. The system according to claim 14, wherein: To train the neural network model, the one or more processors execute the instructions to: The HR residual image corresponding to the training image is divided to generate the plurality of HR residual sub-images corresponding to the training image.

16. A computer-readable medium, characterized in that Computer instructions are stored for increasing the image resolution of a digital image, wherein when the instructions are executed by one or more processors, the one or more processors are caused to perform the steps comprising: performing bicubic upsampling on the digital image to generate a base high-resolution (HR) image; Converting the digital image from a red-green-blue (RGB) color space to a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image; Converting the LR residual image into a plurality of HR residual sub-images corresponding to the digital image using a plurality of convolutional layers of a neural network model; Performing pixel shifting on the multiple HR residual sub-images to generate an HR residual image; The base HR image and the HR residual image are combined to generate an HR image corresponding to the digital image.

17. The computer-readable medium of claim 16, wherein: The instructions further cause the one or more processors to perform steps including: The neural network model is trained using a plurality of training image pairs, each of the plurality of training image pairs comprising: an LR image corresponding to the training image, the LR image having degraded image quality and configured as an input to the neural network model; A plurality of HR residual sub-images corresponding to the training images and configured as target outputs of the neural network model.

18. The computer-readable medium of claim 17, wherein: The instructions further cause the one or more processors to perform steps including: filtering the training image using a low-pass filter to generate a filtered image; Downsampling the filtered image to generate a downsampled LR image; The image quality of the downsampled LR image is degraded by adding noise and artifacts to generate the LR image corresponding to the training image.

19. The computer-readable medium of claim 18, wherein: The instructions further cause the one or more processors to perform the following steps: applying an unbalanced unsharp mask to the training image to generate a contrast enhanced image; subtracting an upsampled version of the downsampled LR image from the contrast-enhanced image to generate a HR residual image corresponding to the training image; The HR residual image corresponding to the training image is divided to generate the plurality of HR residual sub-images corresponding to the training image.

20. An image resolution adjustment system for improving the image resolution of a digital image, characterized in that: The system comprises: an upsampling module, configured to perform bicubic upsampling on the digital image to generate a basic high-resolution (HR) image; a color space processing module for converting the digital image from a red-green-blue (RGB) color space into a luminance (Y), chrominance blue difference (Cb), and chrominance red difference (Cr) (YCbCr) color space to generate a low-resolution (LR) residual image; a convolution module, configured to convert the LR residual image into a plurality of HR residual sub-images corresponding to the digital image; a pixel shift module, configured to perform pixel shifting on the plurality of HR residual sub-images to generate an HR residual image; An additional module is provided for combining the basic HR image and the HR residual image to generate an HR image corresponding to the digital image.

Images