Dynamic image resolution assessment

Abstract

A computer-implemented method for image resolution assessment of a digital image comprises: extracting a plurality of image blocks from the digital image according to a predefined sampling pattern; applying a Gaussian window to each of the plurality of image blocks to generate a plurality of processed image blocks; determining a two-dimensional (2D) energy spectrum of each of the plurality of processed image blocks by discrete Fourier transformation (DFT). Using the 2D energy spectrum, determining a one-dimensional (1D) energy spectrum of each of the plurality of processed image blocks; applying a threshold to the 1D energy spectrum to obtain an image resolution of each of the plurality of processed image blocks; and adjusting a video characteristic of the digital image according to a maximum image resolution of a plurality of image resolutions determined for the plurality of processed image blocks.

Description

Technical Field

[0001] The present invention relates to dynamic image resolution evaluation before deep neural network (DNN) processing of image data. The image data may be image data in real-time streaming video, etc. Background Art

[0002] Deep neural networks (DNNs) are trained on specialized datasets to perform specific image processing tasks, such as image recognition. In addition to the training data, DNNs typically perform well on test datasets. However, when the same DNN model is deployed on a mobile device (e.g., a smartphone), the accuracy of the DNN is found to be significantly reduced. The root of this problem lies in the difference between the high-quality sample videos in the test dataset (as well as the DNN training dataset) and the low-quality streaming videos typically associated with the use of mobile devices. Among all factors, the actual low resolution of the video frames is a major reason that affects the performance of DNNs. This low resolution is defined in terms of the highest frequency that can be represented in a digital image.

[0003] When the resolution is low, the image looks blurry and fails to activate the low-level convolution kernels of the DNN, which are trained on sharp images for feature extraction, e.g., to determine edges and corners. The weak activation of low-level kernels leads to ineffective inference of high-level kernels, which causes a chain reaction that ultimately reduces the accuracy of DNNs deployed on mobile devices. Low resolution is often associated with streaming videos shot with old devices and low-end cameras, over-compression (especially in low-bandwidth settings), and fake high resolution (HR) videos intentionally created by Internet users through upsampling. Therefore, evaluating image resolution on the mobile device side for use with DNNs is an important task to improve DNN performance. Summary of the invention

[0004] Various examples are now described to introduce a set of concepts in a simplified form that are further described below in the Detailed Description. This Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] According to a first aspect of the present invention, a computer-implemented method for image resolution assessment of a digital image is provided. The method comprises: extracting a plurality of image blocks from the digital image according to a predefined sampling pattern; applying a Gaussian window to each of the plurality of image blocks to generate a plurality of processed image blocks; determining a two-dimensional (2D) energy spectrum of each of the plurality of processed image blocks by discrete Fourier transformation (DFT); determining a one-dimensional (1D) energy spectrum of each of the plurality of processed image blocks using the determined 2D energy spectrum; obtaining an image resolution of each of the plurality of processed image blocks by applying a threshold to the determined 1D energy spectrum; and adjusting the video characteristics of the digital image according to a maximum image resolution of the plurality of image resolutions determined for the plurality of processed image blocks.

[0006] According to the first aspect, in a first implementation of the method, the predefined sampling pattern is selected from a plurality of predefined sampling patterns according to a received input.

[0007] According to the first aspect itself or any of the above-mentioned implementations of the first aspect, in a second implementation of the method, the predefined sampling pattern is one of the following patterns: a five-point diamond pattern, wherein a total of 5 image blocks are selected from the intersections of the 3 vertical axes and 3 horizontal axes of the digital image as the multiple image blocks; a nine-point diamond pattern, wherein a total of 9 image blocks are selected from the intersections of the 5 vertical axes and 5 horizontal axes of the digital image as the multiple image blocks; a nine-point square pattern, wherein a total of 9 image blocks are selected from the intersections of the 3 vertical axes and 3 horizontal axes of the digital image as the multiple image blocks; a thirteen-point square pattern, wherein a total of 13 image blocks are selected from the intersections of the 5 vertical axes and 5 horizontal axes of the digital image as the multiple image blocks.

[0008] According to the first aspect itself or any of the above-mentioned implementations of the first aspect, in a third implementation of the method, determining the 1D energy spectrum of each of the multiple processed image blocks includes: accumulating the determined 2D energy spectrum along multiple directions between multiple pixels within each processed image block.

[0009] According to the first aspect itself or any of the above-mentioned implementations of the first aspect, in a fourth implementation of the method, determining the 1D energy spectrum of each processed image block among the multiple processed image blocks includes: vertically folding the 2D energy spectrum of the processed image block along the x-axis of the coordinate system; and summing the 2D energy values ​​of multiple pixels symmetrical about the x-axis to obtain a 2D vertically folded energy spectrum.

[0010] According to the first aspect itself or any of the above-mentioned implementation manners of the first aspect, in a fifth implementation manner of the method, determining the 1D energy spectrum of each processed image block among the multiple processed image blocks also includes: horizontally folding the 2D vertically folded energy spectrum of the processed image block along the y-axis of the coordinate system; summing the 2D energy values ​​of multiple pixels symmetrical about the y-axis to obtain the 2D horizontally folded energy spectrum; and determining the 1D energy spectrum of the processed image block by accumulating the 2D horizontally folded energy spectrum along multiple directions between multiple pixels within the processed image block among the multiple processed image blocks.

[0011] According to the first aspect itself or any of the above-mentioned implementations of the first aspect, in a sixth implementation of the method, adjusting the video characteristics of the digital image includes: downsampling the video stream including the digital image according to the maximum image resolution among the multiple image resolutions determined for the multiple processed image blocks.

[0012] According to the first aspect itself or any of the above-mentioned implementations of the first aspect, in a seventh implementation of the method, for each processed image block among the multiple processed image blocks, determining the 2D energy spectrum, determining the 1D energy spectrum, and applying a threshold to the determined 1D energy spectrum to obtain the image resolution are performed in parallel.

[0013] According to a second aspect of the present invention, a system is provided. The system includes a memory storing instructions and one or more processors communicating with the memory. The one or more processors execute the instructions to: extract multiple image blocks from a digital image according to a predefined sampling pattern; apply a Gaussian window to each of the multiple image blocks to generate multiple processed image blocks; determine a 2D energy spectrum of each of the multiple processed image blocks by discrete Fourier transformation (DFT); determine a 1D energy spectrum of each of the multiple processed image blocks using the determined 2D energy spectrum; obtain an image resolution of each of the multiple processed image blocks by applying a threshold to the determined 1D energy spectrum; and adjust the video characteristics of the digital image according to the maximum image resolution of the multiple image resolutions determined for the multiple processed image blocks.

[0014] According to the second aspect, in a first implementation of the system, the one or more processors are further configured to: select the predefined sampling mode from a plurality of predefined sampling modes based on received input.

[0015] According to the second aspect or any of the above-mentioned implementations of the second aspect, in a second implementation of the system, the one or more processors are also used to determine the 1D energy spectrum of each processed image block among the multiple processed image blocks, including: accumulating the determined 2D energy spectrum along multiple directions between multiple pixels within each processed image block.

[0016] According to the second aspect or any of the above-mentioned implementations of the second aspect, in a third implementation of the system, in order to determine the 1D energy spectrum of each processed image block among the multiple processed image blocks, the one or more processors are also used to: vertically fold the 2D energy spectrum of the processed image block along the x-axis of the coordinate system; and sum the 2D energy values ​​of multiple pixels symmetrical about the x-axis to obtain a 2D vertically folded energy spectrum.

[0017] According to the second aspect or any of the above-mentioned implementations of the second aspect, in a fourth implementation of the system, in order to determine the 1D energy spectrum of each processed image block among the multiple processed image blocks, the one or more processors also execute the instructions to: horizontally fold the 2D vertically folded energy spectrum of the processed image block along the y-axis of the coordinate system; sum the 2D energy values ​​of multiple pixels symmetrical about the y-axis to obtain a 2D horizontally folded energy spectrum; and determine the 1D energy spectrum of the processed image block by accumulating the 2D horizontally folded energy spectrum along multiple directions between multiple pixels within the processed image block among the multiple processed image blocks.

[0018] According to the second aspect or any of the above implementations of the second aspect, in a fifth implementation of the system, in order to adjust the video characteristics of the digital image, the one or more processors execute the instructions to: downsample the video stream including the digital image according to the maximum image resolution among the multiple image resolutions determined for the multiple processed image blocks.

[0019] According to the second aspect or any of the above implementations of the second aspect, in a sixth implementation of the system, the one or more processors are further used to: for each processed image block among the multiple processed image blocks, determine the 2D energy spectrum, the 1D energy spectrum and the image resolution in parallel.

[0020] According to a third aspect of the present invention, a non-transitory computer-readable medium is provided, storing computer instructions for image resolution assessment of a digital image. When one or more processors execute the instructions, the instructions cause the one or more processors to perform multiple operations. The multiple operations include: extracting multiple image blocks from the digital image according to a predefined sampling pattern; applying a Gaussian window to each of the multiple image blocks to generate multiple processed image blocks; determining a 2D energy spectrum of each of the multiple processed image blocks by discrete Fourier transformation (DFT); determining a 1D energy spectrum of each of the multiple processed image blocks using the determined 2D energy spectrum; obtaining an image resolution of each of the multiple processed image blocks by applying a threshold to the determined 1D energy spectrum; and adjusting the video characteristics of the digital image according to the maximum image resolution of the multiple image resolutions determined for the multiple processed image blocks.

[0021] According to the third aspect, in a first implementation of the non-transitory computer-readable medium, the plurality of operations further comprises: selecting the predefined sampling mode from a plurality of predefined sampling modes according to a received input.

[0022] According to the third aspect or any of the above-mentioned implementations of the third aspect, in a second implementation of the non-transitory computer-readable medium, in order to determine the 1D energy spectrum of each processed image block among the multiple processed image blocks, the instruction also causes the one or more processors to perform the following steps: accumulating the determined 2D energy spectrum along multiple directions between multiple pixels within each processed image block.

[0023] According to the third aspect or any above-mentioned implementation manner of the third aspect, in a third implementation manner of the non-transitory computer-readable medium, in order to determine the 1D energy spectrum of each processed image block among the multiple processed image blocks, the instruction further enables the one or more processors to perform the following steps: vertically folding the 2D energy spectrum of the processed image block along the x-axis of the coordinate system; summing the 2D energy values ​​of multiple pixels symmetrical about the x-axis to obtain a 2D vertically folded energy spectrum.

[0024] According to the third aspect or any above-mentioned implementation manner of the third aspect, in a third implementation manner of the non-transitory computer-readable medium, in order to determine the 1D energy spectrum of each processed image block among the multiple processed image blocks, the instruction also enables the one or more processors to perform the following steps: horizontally folding the 2D vertically folded energy spectrum of the processed image block along the y-axis of the coordinate system; summing the 2D energy values ​​of multiple pixels symmetrical about the y-axis to obtain a 2D horizontally folded energy spectrum; and determining the 1D energy spectrum of the processed image block by accumulating the 2D horizontally folded energy spectrum along multiple directions between multiple pixels within the processed image block among the multiple processed image blocks.

[0025] Any of the above examples may be combined with any one or more of the other examples described above to create new embodiments within the scope of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In the drawings, which are not necessarily drawn to scale, like numerals may describe similar components in different views.The drawings generally illustrate, by way of example and not limitation, various embodiments discussed herein.

[0027] Figure 1A block diagram for using a deep learning (DL) training architecture (DLTA) to train a DL program and adjust data or a neural network model based on image resolution is provided for some exemplary embodiments.

[0028] Figure 2 A schematic diagram of generating a trained DL program using a neural network model trained within a DLTA is provided for some exemplary embodiments.

[0029] Figure 3 A schematic diagram of image resolution estimation by parallel processing of multiple image blocks is provided for some exemplary embodiments.

[0030] Figure 4 Schematic diagram of sparse application of a Gaussian window on an image patch in connection with image resolution assessment provided for some exemplary embodiments.

[0031] Figure 5 A schematic diagram of different sampling modes that may be used to select image blocks within an image in connection with image resolution assessment is provided for some example embodiments.

[0032] Figure 6 Some exemplary embodiments provide for image replication during a discrete Fourier transform process in connection with image resolution assessment.

[0033] Figure 7 An “origami” technique for accelerating one-dimensional (1D) omnidirectional spectrum computation in connection with image resolution assessment provided by some exemplary embodiments is shown.

[0034] Figure 8 A flow chart of image resolution assessment is provided for some exemplary embodiments.

[0035] Fig. 9 A block diagram of a representative software architecture is provided for some exemplary embodiments, which may be used in conjunction with the various device hardware described herein.

[0036] Fig.10 A block diagram of a circuit for implementing an algorithm and a device for executing a method is provided for some exemplary embodiments. DETAILED DESCRIPTION

[0037] First, it should be understood that although the following provides an illustrative implementation of one or more embodiments, Figures 1 to 10The disclosed systems and methods described may be implemented using any number of techniques, whether currently known or not yet in existence. The present invention should in no way be limited to the illustrative implementations, drawings, and techniques described below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0038] In the following description, reference is made to the accompanying drawings which form a part of the present invention and which show by way of illustration specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the subject matter of the present invention, it being understood that other embodiments may be utilized and that structural, logical, and electrical variations may be made without departing from the scope of the present invention. Therefore, the following description of the exemplary embodiments is not intended to be limiting, and the scope of the present invention is defined by the appended claims.

[0039] Image resolution assessment has been previously studied in the larger topic of blind or reference-free image quality assessment. To assess the resolution of a digital image, traditional methods compute the spectrum of the digital image via a discrete Fourier transform (DFT) or discrete cosine transform (DCT), and then examine the spectrum distribution to find a quantitative measure of the image resolution. However, DFT and DCT are global transforms applied to the entire image, so the image resolution calculated by such algorithms is usually a coarse measure averaged over all parts of the image. This coarse measure is particularly inaccurate for images with large areas of simple or blurred backgrounds.

[0040] To improve accuracy, some existing algorithms use wavelet transforms for image resolution estimation. Although wavelet transforms can evaluate multi-scale frequency characteristics at each location, the computational complexity of wavelet transforms is usually too high to be used in real-time applications, for example, when DNNs are deployed on mobile devices for real-time image analysis. Other studies focus on analyzing blur indicators that are consistent with human perception. Although these research topics have attracted many users, perceptual consistency is of little help to DNN algorithms associated with additional computational costs. In this regard, existing resolution estimation algorithms do not provide techniques for fast resolution estimation (e.g., resolution estimation that can be performed in milliseconds on mobile devices) nor do they produce reasonably accurate resolution assessment indicators.

[0041] The techniques disclosed herein can be used to address the effects of image blur associated with existing resolution assessment algorithms. More specifically, the techniques disclosed herein can be used to dynamically assess image resolution before video data (e.g., real-time streaming video) is subjected to DNN processing (e.g., by a DNN application executed on a computing device). The image resolution assessment techniques disclosed herein use a sparse windowed Fourier transform to analyze the local frequency characteristics of a digital image (e.g., an image in a video stream). Several different image sampling modes can be used to sample a digital image and select a fixed-size image chip (e.g., a rectangular image portion) from the digital image for analysis. The term "image chip" as used herein refers to an image portion (e.g., a rectangular image portion) of a digital image.

[0042] The image blocks are multiplied by a Gaussian window that keeps the centers of the image blocks unchanged and blurs the image blocks away from their centers. The application of the Gaussian window avoids interference caused by strong high-frequency signals generated by the boundaries of the image blocks. Then, the DFT is applied to each image block to calculate the 2D energy spectrum of each image block. The 2D energy spectrum represents the energy distribution of various frequencies along various directions. By accumulating 2D energies with the same polar radius, the energy in the 2D space is projected onto the 1D omnidirectional frequency space to obtain the 1D energy distribution. This process is accelerated by folding the 2D energy spectrum vertically and then horizontally using the "origami" technique discussed in this article. Next, a single indicator of image resolution is obtained at each sampling position (i.e., for each image block) by thresholding the 1D energy distribution. The final indicator of image resolution is calculated as the maximum value of multiple image resolution indicators calculated for all sampling positions (or image blocks), and its value range is [0, 1). Because DFT can be computed on small scales and spectral analysis of different regions can be performed in parallel, the disclosed technique can be performed on multiple image blocks simultaneously (to more accurately assess image resolution) to capture clear features that may appear in various parts of the image. The final image resolution assessment can be performed within milliseconds, regardless of the image size.

[0043] Since blurry videos and images are common on the Internet, it is important to develop image resolution estimation techniques so that DNNs can be deployed on mobile devices to efficiently process real-time data streams. The following are unique features of the currently disclosed image resolution estimation techniques that are not available in the current (prior art) technology: (a) fast image resolution estimation based on parallel sparse windowed Fourier transform, where the parallel sparse windowed Fourier transform is applied to multiple image patches selected by one of a set of unique sampling patterns; (b) four sampling patterns for image sampling, where these sampling patterns can be used for image resolution estimation; (c) an "origami" technique for determining 1D omnidirectional energy spectrum, where the 2D energy spectrum is folded horizontally and vertically to significantly reduce the 1D energy spectrum computation and provide an efficient image resolution determination method to implement DNN processing on the mobile device side.

[0044] Figure 1 A block diagram 100 of using a deep learning (DL) training architecture (DLTA) to train a DL program 110 or data or neural network model adjustment based on image resolution is provided for some exemplary embodiments. In some exemplary embodiments, machine-learning programs (MLPs), including deep learning programs, also collectively referred to as machine learning algorithms or tools, are used to perform operations related to associating data or other artificial intelligence (AI)-based functions.

[0045] like Figure 1 As shown, deep learning program training 108 can be performed within a deep learning training architecture (DLTA) 106 based on training data 102 (which may include features). During deep learning program training 108, features in the training data 102 can be evaluated in order to further train the DL program. The DL program training 108 produces a trained DL program 110, which can include one or more classifiers 112, which can be used to provide evaluation results 116 based on new data 114.

[0046] Deep learning is a subset of machine learning, a field of study that gives computers the ability to learn without being explicitly programmed. Machine learning explores the study and construction of algorithms, also referred to herein as tools, that can learn from existing data, relate data to each other, and make predictions about new data. Such machine learning tools work by building a model based on exemplary training data (e.g., 102) in order to make data-driven predictions or decisions represented by outputs or deep learning program evaluation results 116. Although exemplary embodiments are presented for a few machine learning tools (e.g., deep learning training architectures), the principles presented herein may also be applied to other machine learning tools.

[0047] In some exemplary embodiments, different machine learning tools may be used. For example, Logistic Regression (LR), Naive-Bayes, Random Forest (RF), neural network (NN), matrix factorization, and Support Vector Machine (SVM) tools may be used during program training 108 (e.g., to correlate training data 102).

[0048] Two common problems in machine learning are classification problems and regression problems. A classification problem is to classify an item into one of several class values ​​(e.g., is this object an apple or an orange?). The goal of a regression algorithm is to quantify some items (e.g., by providing real values). In some embodiments, DLTA 106 can be used to use machine learning algorithms that use training data 102 to find correlations between identified features that affect the results.

[0049] The machine learning algorithm analyzes new data 114 using features in the training data 102 to generate an evaluation result 116. These features include various measurable properties of the phenomenon being observed that are used to train the ML program. The concept of features is related to the concept of explanatory variables used in statistical techniques such as linear regression. Selecting informative, discriminative, and independent features is very important for the effective operation of MLP in pattern recognition, classification, and regression. Features can be of different types, for example, numeric features, strings, and graphics. In some aspects, the training data can be of different types, with numeric features, for use by a computing device.

[0050] In some aspects, the features used during the DL program training 108 may include one or more of the following data: sensor data from multiple sensors (e.g., audio sensors, motion sensors, image sensors), actuator event data from multiple actuators (e.g., wireless switches or other actuators), external information sources from multiple external sources, timer data related to sensor state data (e.g., time sensor data is obtained), actuator event data or external information source data, user communication information, user data, user behavior data, etc.

[0051] The machine learning algorithm uses the training data 102 to find correlations between the identified features that affect the evaluation results 116. In some exemplary embodiments, the training data 102 includes labeled data, which is known data of one or more identified features and one or more results. Based on the training data 102 (which may include the identified features), the DL program is trained using DL program training 108 within the DLTA 106. The training result is a trained DL program 110. When the DL program 110 is used to perform an evaluation, the new data 114 is provided as input to the trained DL program 110, and the DL program 110 generates an evaluation result 116 as output.

[0052] In some aspects, the new data 114 (which may include real-time streaming data) may be processed using one or more techniques disclosed herein to perform image resolution assessment 118. Image data or neural network model adjustments 120 may be performed based on the assessed image resolution. For example, if the image resolution associated with the new data 114 is assessed to be equal to or less than a resolution threshold, one or more video characteristics of the new data 114 may be adjusted to match the video characteristics used to train the DL program 110. In some aspects, the video characteristic adjustments may include downsampling the new data 114 to produce a clearer image. In other aspects, the neural network model used by the DLTA 106 may be retrained using training data of a lower resolution that matches the assessed resolution of the new data so that the model is better suited to process video data having the assessed resolution.

[0053] The term "image resolution" used in this article refers to the image resolution determined according to the sampling frequency of the image. According to the Nyquist-Shannon sampling theorem, for a given sampling rate fs of a discrete signal processing system, if the original signal does not include a sinusoidal component at (or greater than) the Nyquist frequency B = fs / 2, an accurate reconstructed signal can be guaranteed. Therefore, the Nyquist frequency B is the upper limit of the sampling frequency that can be represented by a digital image. A digital image sampled from a two-dimensional (2D) continuous signal at the highest image sampling frequency f (where f = B) is called a "full-resolution image". A digital image sampled from a 2D continuous signal at the highest image sampling frequency f = B / 2 is called a "half-resolution image". Therefore, the resolution of a digital image can be defined as r = f / B, where r∈[0,1), f represents the image sampling frequency, and B represents the Nyquist frequency. In this context, image resolution is distinguished from image size, which is represented by the number of pixels.

[0054] Figure 2 A schematic diagram 200 of generating a trained DL program 206 using a neural network model 204 trained within a DLTA 106 is provided for some example embodiments. Figure 2 , source data 202 may be analyzed by a neural network model 204 (or another machine learning algorithm or technique) to generate a trained DL program 206 (which may be the same as the trained DL program 110). Source data 202 may include a training data set (e.g., 102), including data identified by one or more features. The terms "neural network" and "neural network model" as used herein may be interchangeable.

[0055] Machine learning techniques are used to train models to accurately predict data that is input into these models (e.g., what a user said in a given utterance, whether a noun is a person, place, or thing, what will the weather be like tomorrow). During the learning phase, these models are developed based on the input training data set to optimize these models to correctly predict the output corresponding to the given input. In general, the learning phase can be supervised, semi-supervised, or unsupervised, accounting for the decrease in the level of providing the "correct" output corresponding to the training input. During the supervised learning phase, all outputs are provided to the model and the model is guided to develop general rules or algorithms that map inputs to outputs. In contrast, during the unsupervised learning phase, the input does not provide the expected output, so that the model can develop its own rules to discover relationships within the training data set. During the semi-supervised learning phase, an incompletely labeled training set is provided, and some outputs of the training data set are known and some are unknown.

[0056] The model can be run on the training dataset for several cycles (epochs), where the training dataset is repeatedly input into the model to refine its results (i.e., the entire dataset is processed in one cycle). In the iterative process, the model (e.g., a neural network model or another machine learning model) is run on (a portion of) the entire dataset in mini batches. In the supervised learning phase, a model is developed to predict the output corresponding to a given input set (e.g., source data 202), and this model is evaluated in several cycles to provide a more reliable specification of the output corresponding to the given input, which requires the maximum number of inputs in the training dataset to satisfy all the data in the training set. In another example, for the unsupervised learning phase, a model is developed to cluster the dataset into n groups, and the consistency of the model placing a given input into a given group and the reliability of the model generating n expected clusters in each cycle are evaluated in several cycles.

[0057] These models are evaluated after running for a cycle, and the values ​​of the variables of these models (e.g., weights, biases, or other parameters) are adjusted in an attempt to better refine the models in an iterative manner. The term "weights" used in this article refers to the parameters used by the machine learning model. During the reverse calculation process, the model can output gradients, which can be used to update the weights associated with the forward calculation. The terms "forward calculation" and "reverse calculation" used in this article refer to calculations performed in conjunction with the training of a neural network model (or another model). During the forward and reverse calculation processes, the calculations performed in the current iteration modify the weights based on the results of the previous iterations (e.g., based on the gradients generated at the end of the previous reverse calculation). In a distributed synchronous training environment for deep neural networks, in order to update the neural network model weights (i.e., gradient synchronization), gradient aggregation, averaging, and distribution between workers can be run sequentially with back propagation (i.e., neural network model layer processing during reverse calculation).

[0058] In various aspects, the evaluation results may be biased in the direction of false negatives, biased in the direction of false positives, or biased on average according to the overall accuracy of the model. Depending on the machine learning technique used, each value may be adjusted in a variety of ways. For example, in a genetic algorithm or evolutionary algorithm, the value of the most successful model in predicting the expected output is used to develop the value used by the model in subsequent cycles, which may include random changes / mutations to provide additional data points. Those of ordinary skill in the art are familiar with several other machine learning algorithms that may be applicable to the present invention, including linear regression, random forest, decision tree learning, neural networks, deep neural networks, etc.

[0059] Each model develops rules or algorithms over several cycles by changing the values ​​of one or more variables that affect the input to more closely match the desired outcome, but perfect accuracy and precision may not be achieved because the training dataset may vary and, at best, be very large. Therefore, the number of cycles that make up the learning phase may be set to a given number of trials or a fixed time / computational budget, or may terminate before a given number / budget is reached when the accuracy of a given model is high enough or low enough or reaches an accuracy plateau. For example, if the training phase is designed to run for n cycles and generate a model with at least 95% accuracy, and such a model is generated before the nth cycle, the learning phase may end early and use the generated model that meets the final target accuracy threshold. Similarly, if the accuracy of a given model is not high enough to meet a random chance threshold, the learning phase of that model may terminate early, although other models within the learning phase may continue to train. Similarly, the learning phase of a given model may terminate before the number of cycles / computational budget is reached when the given model continues to provide similar accuracy over multiple cycles or its results are wavering (having reached a performance plateau).

[0060] After the learning phase is complete, the model is finalized. In some exemplary embodiments, the finalized model is evaluated according to a test criterion. In a first example, a test data set including known outputs corresponding to inputs is input into the finalized model to determine the accuracy of the model when processing data that has not been trained. In a second example, the false positive rate or the false negative rate can be used to evaluate the finalized model. In a third example, the delineation between the data clusters in each model is used to select the model that produces the clearest boundaries for its data clusters.

[0061] In some exemplary embodiments, the DL program 206 is trained by a neural network 204 (e.g., a deep learning network, a deep convolutional network, or a recursive neural network), which includes a series of "neurons" arranged into a network, such as Long Short Term Memory (LSTM) nodes. Neurons are architectural elements used for data processing and artificial intelligence (particularly machine learning), which include memory, and the memory can determine when to "remember" and when to "forget" the value stored in the memory based on the weight of the input provided to a given neuron. Each neuron used in this article is used to receive a predefined number of inputs from other neurons in the network in order to provide relevant outputs and sub-relevant outputs for the content of each frame being analyzed. Individual neurons can be linked together and / or organized into a tree structure in various configurations of the neural network to provide interactive and relational learning modeling to determine how each frame in the discourse is related to each other.

[0062] For example, an LSTM as a neuron includes several gates for processing input vectors (e.g., phonemes in speech), storage cells, and output vectors (e.g., context representations). The input gate and output gate control the information flowing into and out of the storage cell, respectively, while the forget gate optionally deletes information from the storage cell based on the input of the previous link unit of the neural network. The weights and bias vectors of the various gates are adjusted during the training phase, and once the training phase is completed, these weights and biases are finally determined for normal operation. Those skilled in the art will understand that neurons and neural networks can be constructed programmatically (e.g., through software instructions) or by linking each neuron to form a dedicated hardware for the neural network.

[0063] Neural networks analyze data using features to generate an assessment result (e.g., recognize a unit of speech). Features are independently measurable properties of the phenomenon being observed. The concept of features is related to the concept of explanatory variables used in statistical techniques such as linear regression. In addition, deep features represent the output of nodes on the hidden layer of a deep neural network.

[0064] A neural network (e.g., 204), sometimes referred to as an artificial neural network or neural network model, is a computer system that is based on the biological neural networks of animal brains. These systems gradually improve in performance (called learning) to perform tasks, typically without the need for task-specific programming. For example, in image recognition, a neural network can be taught to recognize images that include an object by analyzing example images that have been labeled with the name of the object, and the neural network, after learning the object and name, can use the analysis results to recognize the object in unlabeled images. Neural networks are based on a set of connected units called neurons, where each connection between neurons, called a synapse, can transmit a unidirectional signal whose activation strength varies with the strength of each connection. A receiving neuron can activate and propagate a signal to downstream neurons connected to it, typically based on whether the combined input signal from potentially many transmitting neurons is of sufficient strength, where strength is a parameter.

[0065] A deep neural network (DNN) is a stacked neural network consisting of multiple layers. These layers are made up of nodes, which are locations where computations occur, scattered across neurons in the human brain that fire when they are stimulated enough. A node combines the input of data with a set of coefficients or weights that can amplify or suppress that input, which assigns importance to the input for the task the algorithm is trying to learn. These input-weight products are summed and the sum is passed through the node's activation function to determine whether and to what extent the signal passes further through the network to affect the final result. DNNs use cascaded layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output of the previous layer as input. High-level features are derived from low-level features to form a hierarchical representation. The layers following the input layer can be convolutional layers, which produce feature maps that are filtered results of the input and used by the next convolutional layer.

[0066] In the training of DNN architectures, regression is constructed as a set of statistical procedures for estimating the relationship between variables, which may include the minimization of a cost function. The cost function may be implemented as a function that returns a number representing the performance of the neural network in mapping the training examples to the correct output. During training, if the cost function value is not within a predefined range, backpropagation is used based on known training images, where backpropagation is a common method for training artificial neural networks, which are used with optimization methods such as the stochastic gradient descent (SGD) method.

[0067] The use of backpropagation can include both propagation and weight updates. When an input is given to a neural network, that input is propagated forward through the neural network layer by layer until that input reaches the output layer. The output of the neural network is then compared to the desired output using a cost function, and an error value is calculated for each node in the output layer. Starting from the output, these error values ​​are propagated backwards until each node has an associated error value that roughly represents its contribution to the original output. Backpropagation can use these error values ​​to calculate the gradient of the cost function with respect to the weights in the neural network. The calculated gradients are input into the selected optimization method to update the weights in an attempt to minimize the cost function.

[0068] Even though the training architecture 106 is referred to as a deep learning training architecture using a neural network model (and the trained program is referred to as a trained deep learning program, such as 110 or 206), the present invention is not limited thereto, and other types of machine learning training architectures may be used for model training.

[0069] In some aspects, a deep learning program (e.g., 110 or 206) can be used in conjunction with video data processing (e.g., object recognition including face recognition) and can be retrained based on real-time estimated image resolution data using one or more of the techniques disclosed herein. In some aspects, retraining of the deep learning program or adjustment of video characteristics of the input video data (e.g., downsampling) can be performed based on different thresholds associated with the estimated image resolution.

[0070] Figure 3 A schematic diagram of image resolution estimation 300 by parallel processing of multiple image blocks is provided for some exemplary embodiments. Figure 3 , the image resolution assessment 300 can be dynamically performed on a mobile device (e.g., when the mobile device is receiving real-time video data) to assess the resolution of one or more images in the video data. The image resolution assessment can include processing chains 314 to 322, each of which applies a Gaussian window to an input image block (e.g., 304 to 312) to obtain a processed image block (e.g., 324 to 332), a two-dimensional (2D) energy spectrum (e.g., 334 to 342), and a 1D omnidirectional energy spectrum (e.g., 344 to 352). The final resolution assessment result 374 is obtained based on the 1D omnidirectional energy spectra 344 to 352, and the details will be provided below.

[0071] For example, image 302 may be part of a data stream received by a computing device that performs image processing using a DNN. A sampling pattern may be applied to image 302 to obtain image blocks 304, 306, 308, and 310. Figure 3 In the particular example shown, a five-point diamond sampling pattern is applied, but other sampling patterns may also be used (e.g., Figure 5 Even though the image blocks 304 to 312 may be obtained from a color image, in some aspects, the entire image 302 (or image block) may be converted to a black and white image (or image block) before starting the processing chain 314, 316, 318, 320, and 322 corresponding to the image block.

[0072] In some aspects, each of the image blocks 304 to 312 is shaped as a rectangle (e.g., a square) of a predefined size. In the initial processing step of each of the processing chains 314 to 322, each of the image blocks 304 to 312 is multiplied element by element by a Gaussian window to obtain processed image blocks 324, 326, 328, 330, and 332, respectively, wherein the Gaussian window is 1 in the middle and the value decreases away from the center. The application of the Gaussian window keeps the center of the image block unchanged, moves the image blocks away from their center, and avoids interference caused by strong high-frequency signals generated by the boundaries of the image blocks. In addition, since the Gaussian window can only be applied to a limited number of image blocks and cannot be applied to the entire image, this application can be referred to as a sparse application of the Gaussian window.

[0073] Figure 4 A schematic diagram 400 of a sparse application of a Gaussian window to an image patch in connection with image resolution estimation is provided for some exemplary embodiments. Figure 4 , shows a sparse application of a Gaussian window to nine image blocks within a digital image, resulting in a representation 402 of the processed image blocks after the Gaussian window application.

[0074] Figure 5 Schematic diagrams of different sampling modes 502, 504, 506, and 508 that may be used to select a plurality of image blocks within an image in connection with image resolution assessment are provided for some exemplary embodiments.

[0075] Due to the limited aperture of the camera, the image is usually clear in the focus area and has varying degrees of blur in other areas. When taking photos / videos, photographers usually focus the lens on the main object and place these main objects in the central area of ​​the picture (frame). Based on these observations, sampling modes 502, 504, 506 and 508 can be used in combination with the image resolution assessment technology disclosed in this article. Sampling modes 502 to 508 are designed to capture important image objects using a minimum number of samples (or image blocks). According to streaming video analysis, the most common locations for image objects are around the center of the image and 1 / 3 of the left, right, top and bottom of the image. These areas are fully covered by the nine-point diamond pattern and the thirteen-point square pattern, as shown in image patterns 504 (with image block 512) and 508 (with image block 516).

[0076] When the computational budget is extremely low, the image sampling scheme can be reduced to a five-point diamond and a nine-point square, as shown in image patterns 502 (with image block 510) and 506 (with image block 514). The peripheral sampling points are not distributed at the center and 1 / 3 (image height and width) positions, but are placed around the 1 / 4 (image height and width) positions. The 1 / 4 scheme has image blocks that are more evenly distributed over the image and cover more area than the 1 / 3 scheme. In addition, many images include edges, corners, and other sharp features around the 1 / 4 position.

[0077] In some aspects, the image blocks 510, 512, 514, and 516 are all of the same shape, which is a square. In some aspects, the position of each of the image blocks 510, 512, 514, and 516 can be determined based on the image height (e.g., h pixels) and the image width (e.g., w pixels), such as Figure 5 In some aspects, since the Fourier transform can be used in conjunction with image resolution assessment, and since the Fourier transform is particularly efficient when the input image size is a product of multiples of 2, 3, 5, and combinations thereof, the image block size can be set to 64, 81, 125, 128, etc., depending on the actual application. In some aspects, some image blocks can overlap each other.

[0078] Reference again Figure 3 After applying the Gaussian window and generating processed image blocks 324, 326, 328, 330, 332 in processing chains 314, 316, 318, 320, and 322, respectively, a digital Fourier transform (DFT) (e.g., a fast Fourier transform (FFT)) is applied to these processed image blocks to calculate 2D energy spectra 334, 336, 338, 340, and 342, respectively. The 2D energy spectrum represents the energy distribution of various frequencies along different directions in each processed image block.

[0079] The DFT application assumes that the image size is infinite and that the image consists of vertical and horizontal copies of the input image block, such as Figure 6 shown.

[0080] Figure 6 600 is shown in a discrete Fourier transform process related to image resolution assessment provided by some exemplary embodiments. More specifically, before DFT is applied, the image block 602 can be replicated multiple times in the horizontal and vertical directions, such as Figure 6 shown.

[0081] Through replication, artificial vertical and horizontal edges are generated along the image block boundaries. Artificial edges can generate strong pseudo-energy in the high-frequency region of the frequency domain, which in many cases can overwhelm the real image energy.

[0082] In some aspects, applying the Gaussian window to image blocks 304 to 312 may be expressed as follows:

[0083] I win (i, j) = I(i, j) · G(i, j), where if the image block size is k×k pixels, then

[0084] The application of the Gaussian window has a property that the Fourier transform of a Gaussian function is another Gaussian function, which acts as a low-pass filter in the frequency domain, blurring the spectrum slightly without interfering with the spectral analysis. After applying the Gaussian window to each image block, a 2D FFT is performed on each image block to determine the 2D energy spectrum of each image block (e.g., F(u, v) at position (u, v)), which has a real number at each pixel. The logarithmic transform E(u, v) of the 2D energy spectrum F(u, v) is determined as follows to brighten high-frequency signals: E(u, v) = 10·log 10 F(u,v) / 0.00005°

[0085] By accumulating energy with the same polar radius, the 2D energy spectra 334 to 342 of the image blocks 304 to 312 are projected into the 1D frequency space to obtain 1D omnidirectional energy spectra 344, 346, 348, 350 and 352 of the image blocks 304 to 312, respectively.

[0086] The value of a point in the energy spectrum E(u, v) on (u, v) represents the frequency Along direction The energy of. When it comes to image resolution, the direction is not important. In this regard, the 2D energy spectrum is converted to a 1D omnidirectional energy spectrum (e.g., representing 344 to 352) by accumulating the energy in different directions (i.e., omnidirectional accumulation), as shown below: Wherein, E(f) represents the 1D omnidirectional energy spectrum shown in 344 to 352.

[0087] Since the above equation for determining E(f) involves resource-intensive multiplications and square root calculations, the E(f) determination process can be performed faster using an origami technique (e.g., by folding the 2D energy spectrum vertically and summing all corresponding values). Repeating the same origami process horizontally, such as Figure 7 Finally, E(f) can only be efficiently accumulated and determined in one of the four quadrants of the 2D energy spectrum.

[0088] Figure 7 An “origami” technique 700 for accelerating 1D omnidirectional spectrum computation related to image resolution assessment provided by some exemplary embodiments is shown. Figure 7 , the 2D energy spectrum 702 calculated for an image block can be vertically folded across the horizontal axis (or x-axis) 704 to obtain a 2D energy spectrum 706. During the folding process, the 2D energy values ​​of multiple pixels symmetrical along the x-axis 704 are summed, and the summed value is represented in the 2D energy spectrum 706. The "origami" technique 700 continues to fold the 2D energy spectrum 706 horizontally across the vertical axis (or y-axis) 708 to obtain a 2D energy spectrum 701 (one quarter the size of the 2D energy spectrum 702). During the folding process along the y-axis 708, the 2D energy values ​​of multiple pixels symmetrical along the y-axis 708 are summed, and the summed value is represented in the 2D energy spectrum 701. Then, the 1D omnidirectional energy spectrum calculation can be performed on the reduced-size 2D energy spectrum 701 (rather than the full-size 2D energy spectrum 702), making the image resolution assessment faster.

[0089] Reference again Figure 3 , the 1D omnidirectional energy spectra 344 to 352 are all based on the relationship between the image energy in the frequency domain (measured in decibels (or dB) along the y-axis) and the image resolution (expressed as a value between 0 and 1 along the x-axis). The 1D omnidirectional energy spectra can be limited by applying the corresponding thresholds 354, 356, 358, 360 and 362 when the image energy is 0 dB. More specifically, the thresholds 354 to 362 can be used to cut the omnidirectional energy spectrum, and the cut-off points correspond to the positions where the image energy is 0 dB. The image resolution values ​​364, 366, 368, 370, 372 corresponding to the image blocks 304 to 312 are obtained as the intersection of each threshold with the x-axis of the 1D omnidirectional energy spectrum 344 to 352. The final image resolution 374 is determined as the maximum value of all the image resolution values ​​364 to 372 calculated for the image blocks 304 to 312, wherein the final image resolution 374 has a value range of [0,1).

[0090] In some aspects, the processing chains 314 to 322 can be executed in parallel after determining the image blocks according to the sampling pattern. Since the Fourier analysis is performed on different regions, the disclosed image resolution technique is accurate. In addition, since the DFT is very effective at small scales and the spectral analysis on different regions is performed in parallel (e.g., the processing chains 314 to 322), the image resolution assessment technique disclosed herein can be executed within milliseconds, regardless of the image size.

[0091] After determining the final data image resolution, the neural network processing of the image can be optimized by retraining the neural network model or changing one or more characteristics of the video stream with the image. For example, if it is determined that the image resolution is less than a threshold, the video stream can be downsampled to produce a clearer image so that a neural network trained on the high-resolution image can still be applied. Alternatively, the neural network model can be retrained based on the final resolution obtained for the image in the video stream.

[0092] In some aspects, if the resolution of an image / video is estimated using the techniques disclosed herein, the image / video can be downsampled without information loss to save storage space and reduce streaming costs. Some network services require uploaded images / videos to have a minimum resolution for quality control. In this case, the techniques disclosed herein can be used to detect the resolution of an image / video and filter out "fake" high-resolution images / videos that are intentionally upsampled (i.e., images / videos that are not high in resolution but have been upsampled to produce a high-resolution impression). Existing images / videos can also be classified in a library / database based on the actual resolution detected using the disclosed techniques.

[0093] Figure 8 A flowchart of image resolution evaluation is provided for some exemplary embodiments. Method 800 includes operations 802, 804, 806, 808, 810, and 812. By way of example and not limitation, method 800 is described as being performed by a resolution evaluation module in a computing device, for example, by Fig.10 The resolution evaluation module 1060 in the computing device 1000 is executed.

[0094] In operation 802, a plurality of image blocks are extracted from a digital image according to a predefined sampling pattern. For example, the resolution evaluation module 1060 may extract image blocks 304 to 312 from the image 302 using a predefined five-point sampling pattern. In operation 804, a Gaussian window is applied to each of the plurality of image blocks to generate a plurality of processed image blocks. For example, the resolution evaluation module 1060 may apply a Gaussian window to image blocks 304 to 312 to generate processed image blocks 324 to 332, as shown in FIG. Figure 3 shown.

[0095] In operation 806, a 2D energy spectrum of each processed image block in the plurality of processed image blocks is determined by DFT. For example, the 2D energy spectra 334 to 342 of the processed image blocks 324 to 332 are determined by DFT. In operation 808, a 1D energy spectrum of each processed image block in the plurality of processed image blocks is determined using the determined 2D energy spectrum. For example, the resolution evaluation module 1060 determines a 1D omnidirectional energy spectrum 344 to 342 corresponding to the 2D energy spectrum 334 to 342 associated with the image blocks 304-312. In operation 810, the 1D energy spectrum is compared with a threshold value to obtain an image resolution of each processed image block in the plurality of processed image blocks. For example, the resolution evaluation module 1060 applies the threshold values ​​354 to 362 to the 0dB image energy of the 1D omnidirectional energy spectrum to determine the image resolution 364 to 372 of the corresponding image blocks 304 to 312. In operation 812, the video characteristics of the digital image are adjusted according to the maximum image resolution among the plurality of image resolutions determined for the plurality of processed image blocks. For example, the resolution evaluation module 1060 may downsample the image 302 or retrain the neural network model according to the determined image resolution 374. The image resolution 374 is the maximum resolution selected from the image resolutions 364 to 372.

[0096] Fig. 9 A block diagram of a representative software architecture 900 is provided for some exemplary embodiments. The software architecture 900 may be used in conjunction with the various device hardware described herein. Fig. 9 This is only one non-limiting example of software architecture 902, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. Software architecture 902 may be implemented in Fig.10 The system is executed in hardware such as device 1000, where device 1000 includes a processor 1005, a memory 1010, storages 1015 and 1020, and I / O components 1025 and 1030.

[0097] A representative hardware layer 904 is shown, which may represent Fig.10 The representative hardware layer 904 includes one or more processing units 906 with associated executable instructions 908. The executable instructions 908 represent executable instructions of the software architecture 902, including implementing Figures 1 to 8 The hardware layer 904 also includes memory and / or storage modules 910, which also have executable instructions 908. The hardware layer 904 may also include other hardware 912, which represents any other hardware of the hardware layer 904, for example, other hardware shown as part of the device 1000.

[0098] exist Fig. 9In the exemplary architecture of FIG. 902 , the software architecture 902 can be conceptualized as a stack of layers, wherein each layer provides specific functionality. For example, the software architecture 902 can include layers such as an operating system 914, a library 916, a framework / middleware 918, an application 920, and a presentation layer 944. In operation, the application 920 and / or other components within each layer can call an application programming interface (API) call 924 through the software stack, and receive a response, a return value, etc. shown as a message 926 in response to the API call 924. Fig. 9 The layers shown are representative in nature, and not all software architectures 902 have all layers. For example, some mobile or dedicated operating systems may not provide framework / middleware 918, while other operating systems may provide such layers. Other software architectures may include other layers or different layers.

[0099] The operating system 914 can manage hardware resources and provide public services. The operating system 914 may include a kernel 928, services 930, and drivers 932, etc. The kernel 928 may serve as an abstraction layer between the hardware and other software layers. For example, the kernel 928 may be responsible for memory management, processor management (e.g., scheduling), component management, networking, security settings, etc. The services 930 may provide other public services to other software layers. The drivers 932 may be responsible for controlling or connecting to the underlying hardware. For example, depending on the hardware configuration, the drivers 932 may include display drivers, camera drivers, drivers, flash drives, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0100] The library 916 can provide a public infrastructure that can be used by applications 920 and / or other components and / or layers. The function of the library 916 is generally to allow other software modules to perform tasks in an easier way than directly connecting to the underlying operating system 914 functions (e.g., kernel 928, service 930 and / or driver 932). The library 916 may include a system library 934 (e.g., C standard library), which can provide various functions, such as memory allocation functions, string operation functions, and mathematical functions. In addition, the library 916 may include an API library 936, such as a media library (e.g., a library that supports the presentation and operation of various media formats such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.), a graphics library (e.g., an OpenGL framework that can be used to present 2D and 3D graphics content on a display), a database library (e.g., SQLite that can provide various relational database functions), a webpage library (e.g., WebKit that can provide web browsing functions), etc. The library 916 may also include a variety of other libraries 938 to provide many other APIs to the applications 920 and other software components / modules.

[0101] The framework / middleware 918 (sometimes also referred to as middleware) can provide a high-level common infrastructure that can be used by applications 920 and / or other software components / modules. For example, the framework / middleware 918 can provide various graphical user interface (GUI) functions, advanced resource management, advanced location services, etc. The framework / middleware 918 can provide various other APIs that can be used by applications 920 and / or other software components / modules, some of which can be specific to a particular operating system 914 or platform.

[0102] The applications 920 include built-in applications 940, third-party applications 942, and a resolution evaluation module 960. In some aspects, the resolution evaluation module 960 includes suitable circuits, logic, interfaces, and / or codes, and can be used to perform the combined Figure 3 For example, the resolution assessment module 960 can execute the functions within each of the processing chains 314 to 322 in parallel. Even though the resolution assessment module 960 is shown as part of the application 920, the present invention is not limited thereto, and the resolution assessment module 960 can be part of the operating system 914 of the software architecture 902.

[0103] Examples of representative built-in applications 940 may include, but are not limited to, contact applications, browser applications, reader applications, location applications, media applications, communication applications, and / or game applications. Third-party applications 942 may include any of the built-in applications 940 as well as a variety of other applications. In a specific example, third-party applications 942 (e.g., applications developed by entities other than the vendor of a particular platform using Android TM or iOS TM Applications developed with the software development kit (SDK) can be developed on iOS TM 、Android TM , Mobile software running on a mobile operating system such as Phone or other mobile operating systems. In this example, third-party application 942 can call API calls 924 provided by a mobile operating system such as operating system 914 to help implement the functions described herein.

[0104] Applications 920 can utilize built-in operating system functions (e.g., kernel 928, services 930, and drivers 932), libraries (e.g., system libraries 934, API libraries 936, and other libraries 938), and framework / middleware 918 to create a user interface to interact with system users. Alternatively or additionally, in some systems, interaction with the user can be performed through a presentation layer (e.g., presentation layer 944). In these systems, the application / module "logic" can be independent of the aspects of the application / module that interact with the user.

[0105] Some software architectures make use of virtual machines. Fig. 9 In the example of FIG. 1 , a virtual machine is shown by virtual machine 948. A virtual machine creates a software environment in which an application / module can be run as if it were on a hardware machine (e.g., Fig.10 The virtual machine 948 is hosted by a host operating system (e.g., operating system 914) and typically (although not always) has a virtual machine monitor 946. The virtual machine monitor 946 is used to manage the operation of the virtual machine 948 and the connection with the host operating system (i.e., operating system 914). The software architecture 902 is executed in the virtual machine 948, such as the operating system 950, the library 952, the framework / middleware 954, the application 956, and / or the presentation layer 958. These software architecture layers executed in the virtual machine 948 can be the same as the corresponding layers described above, or they can be different.

[0106] Fig.10A block diagram of a circuit for a device implementing an algorithm and execution method is provided for some exemplary embodiments. Not all components need to be used in various embodiments. For example, a client, a server, and a cloud-based network device may each use a different set of components, or, for example, in the case of a server, a larger storage device.

[0107] An exemplary computing device in the form of a computer 1000 (also referred to as computing device 1000, computer system 1000, or computer 1000) may include a processor 1005, memory storage 1010, removable storage 1015, non-removable storage 1020, input interface 1025, output interface 1030, and communication interface 1035, all connected via a bus 1040. Although the exemplary computing device is shown and described as being a computer 1000, the computing device may take different forms in different embodiments.

[0108] Memory storage 1010 may include volatile memory 1045 and nonvolatile memory 1050, and may store programs 1055. Computing device 1000 may include or have access to a computing environment that includes various computer-readable media, such as volatile memory 1045, nonvolatile memory 1050, removable memory 1015, and non-removable memory 1020. Computer storage includes random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other storage technology, compact disc read-only memory (CD ROM), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

[0109] Computer-readable instructions stored in a computer-readable medium (e.g., a program 1055 stored in memory 1010) can be executed by the processor 1005 of the computing device 1000. Hard disk drives, CD-ROMs, and RAM are some examples of articles of manufacture that include non-transitory computer-readable media such as storage devices. The terms "computer-readable medium" and "storage device" do not include carrier waves that are considered too transient. "Computer-readable non-transitory medium" includes all types of computer-readable media, including magnetic storage media, optical storage media, flash memory media, and solid-state storage media. It should be understood that the software can be installed in a computer and sold with the computer. Alternatively, the software can be obtained and loaded into the computer, including obtaining the software through a physical medium or distribution system, including (for example) obtaining the software from a server owned by the software creator or from a server that the software creator does not own but uses. For example, the software can be stored in a server for distribution over the Internet. The terms "computer-readable medium" and "machine-readable medium" used herein are interchangeable.

[0110] The program 1055 may utilize a customer preference structure that utilizes the modules discussed herein, such as the resolution evaluation module 1060. In some aspects, the resolution evaluation module 1060 includes suitable circuits, logic, interfaces and / or code and may be used to perform the combined Figure 3 For example, the resolution assessment module 1060 may perform the functions within each of the processing chains 314 to 322 in parallel, as well as perform the functions in conjunction with the processing chains 314 to 322. Figure 8 Function of discussion.

[0111] Any one or more modules described herein may be implemented using hardware (e.g., a processor of a machine, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or any suitable combination thereof). Moreover, any two or more of these modules may be combined into a single module, and the functionality of a single module described herein may be subdivided between multiple modules. In addition, according to various exemplary embodiments, the modules described herein as being implemented within a single machine, database, or device may be distributed to multiple machines, databases, or devices.

[0112] In some aspects, module 1060 and one or more other modules in program 1055 may be integrated into a single module to perform the corresponding functions of the integrated module.

[0113] Although several embodiments are described in detail above, other modifications may be made. For example, the logic flow described in the figure does not require the specific order or sequential order shown to achieve the desired results. Other steps may be provided, steps may be deleted from the described flow, and other components may be added or removed from the described system. Other embodiments may be within the scope of the appended claims.

[0114] It should also be understood that software including one or more computer executable instructions that facilitate the processes and operations described above in connection with any or all of the steps of the present invention may be installed in and sold with one or more computing devices consistent with the present invention. Alternatively, the software may be acquired and loaded into one or more computing devices, including acquiring the software via physical media or a distribution system, including, for example, acquiring the software from a server owned by the creator of the software or from a server not owned by the creator of the software but used by the creator of the software. For example, the software may be stored in a server for distribution over the Internet.

[0115] In addition, it should be understood by those skilled in the art that the present invention is not limited to the construction and setting details of the components described in the specification or illustrated in the drawings in its application. The embodiments herein can have other embodiments and can be practiced or executed in various ways. In addition, it should be understood that the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of "including", "comprising" or "having" and its variants in this article is intended to cover the items listed thereafter and their equivalents and other items. Unless otherwise limited, the terms "connect", "couple" and "install" and their variants are widely used in this article and cover direct and indirect connections, couplings and installations. In addition, the terms "connect" and "couple" and their variants are not limited to physical or mechanical connections or couplings. In addition, the terms "upper", "lower", "bottom" and "top" are relative and are used to help illustrate, but are not restrictive.

[0116] The components of the illustrative devices, systems, and methods used in accordance with the described embodiments may be implemented at least in part in digital electronic circuitry, analog electronic circuitry, or in computer hardware, firmware, software, or a combination thereof. For example, these components may be implemented as a computer program product (e.g., a computer program, program code, or computer instructions) tangibly embodied in an information carrier, or in a machine-readable storage device, for execution by a data processing apparatus (e.g., a programmable processor, a computer, or multiple computers), or for controlling the operation of a data processing apparatus.

[0117] The computer program can be written in any form of programming language (including compiled or interpreted languages) and can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. The computer program can be deployed to be executed in one computer or in multiple computers at one site, or it can be distributed at multiple sites and interconnected through a communication network. In addition, the functional programs, codes, and code segments used to implement the technology described herein are easily understood by programmers in the field of the technology described herein as being within the scope of the claims. The method steps associated with the illustrative embodiments can be performed by one or more programmable processors, thereby executing computer programs, codes, or instructions to perform functions (e.g., operating on input data and / or generating output). For example, the method steps can also be performed by a dedicated logic circuit (e.g., a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)), and the device for performing the above method can be implemented as the dedicated logic circuit.

[0118] The various illustrative logical blocks, modules, and circuits described in conjunction with the embodiments disclosed herein may be implemented or executed using a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, and optionally, the general purpose processor may also be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in combination with a DSP core, or any other similarly configured combination.

[0119] For example, processors suitable for executing computer programs include general and special microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, the computer also includes one or more mass storage devices (such as magnetic disks, magneto-optical disks, or optical disks) for storing data, or is operably coupled to one or more mass storage devices for storing data to receive data from the mass storage device and / or transfer data to the mass storage device. Information carriers suitable for embodying computer program instructions and data include various forms of non-volatile memory, for example, including semiconductor memory devices, such as electrically programmable read-only memory or electrically programmable ROM (electrically programmable read-only memory, EPROM), electrically erasable programmable ROM (electrically erasable programmable ROM, EEPROM), flash memory devices, data storage disks (e.g., magnetic disks, internal hard disks or removable disks, magneto-optical disks, CD-ROMs and DVD-ROM disks). The processor and memory can be supplemented by or incorporated into a dedicated logic circuit.

[0120] Those skilled in the art will appreciate that any of a variety of different technologies and technical methods may be used to represent information and signals. For example, data, instructions, commands, information, signals, bits, symbols, and image blocks that may be cited in the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or magnetic particles, light fields or light particles, or any combination thereof.

[0121] As used herein, "machine-readable medium (or computer-readable medium)" refers to a device capable of temporarily or permanently storing instructions and data, and may include, but is not limited to, random-access memory (RAM), read-only memory (ROM), buffer memory, flash memory, optical media, magnetic media, cache memory, other types of memory (e.g., Erasable Programmable Read-Only Memory (EEPROM)), and / or any suitable combination thereof. The term "machine-readable medium" should be understood to include a single medium or multiple media (e.g., a centralized or distributed database, or associated caches and servers) that can store processor instructions. The term "machine-readable medium" should also be understood to include any medium or combination of multiple media capable of storing instructions executed by one or more processors 1005, which, when executed by one or more processors 1005, cause one or more processors 1005 to perform any one or more methods described herein. Accordingly, "machine-readable medium" refers to a single storage device or device, as well as a "cloud-based" storage system or storage network comprising multiple storage devices or devices. The term "machine-readable medium" as used herein does not include the signal itself.

[0122] In addition, without departing from the scope of the present invention, the techniques, systems, subsystems and methods described and illustrated as discrete or separate in various embodiments may be combined or integrated with other systems, modules, techniques or methods. Other items shown or described as coupled or directly coupled or communicating with each other may also be coupled or communicated indirectly via an interface, device or intermediate component, either electrically, mechanically or otherwise. Other examples of changes, substitutions and alterations may be determined by those skilled in the art and may be exemplified without departing from the scope disclosed herein.

[0123] Although the present invention has been described with reference to the specific features and embodiments of the present invention, it is apparent that various modifications and combinations of the present invention may be formulated without departing from the present invention. For example, other components may be added to or removed from the described system. Therefore, the specification and the accompanying drawings are only considered as an explanation of the present invention as defined by the appended claims, and are intended to cover any and all modifications, variants, combinations or equivalents falling within the scope of the present invention. Other aspects may be within the scope of the appended claims.

Claims

1. A computer-implemented method for image resolution assessment of digital images, characterized in that The method comprises: extracting a plurality of image blocks from the digital image according to a predefined sampling pattern; applying a Gaussian window to each image block of the plurality of image blocks to generate a plurality of processed image blocks; Determining a two-dimensional (2D) energy spectrum of each processed image block in the plurality of processed image blocks by discrete Fourier transformation (DFT); Determine a one-dimensional (1D) energy spectrum of each processed image block in the plurality of processed image blocks using the determined 2D energy spectrum; obtaining an image resolution of each of the plurality of processed image blocks by applying a threshold to the determined 1D energy spectrum; Video characteristics of the digital image are adjusted based on a maximum image resolution among a plurality of image resolutions determined for the plurality of processed image blocks.

2. The computer-implemented method of claim 1, wherein: The method further comprises: The predefined sampling pattern is selected from a plurality of predefined sampling patterns according to the received input.

3. The computer-implemented method of claim 1, wherein: The predefined sampling mode is one of the following modes: A five-point diamond pattern, wherein a total of five image blocks are selected as the plurality of image blocks from the intersections of the three vertical axes and the three horizontal axes of the digital image; A nine-point diamond pattern, wherein a total of nine image blocks are selected as the plurality of image blocks from the intersections of five vertical axes and five horizontal axes of the digital image; A nine-point square mode, wherein a total of nine image blocks are selected as the plurality of image blocks from the intersections of three vertical axes and three horizontal axes of the digital image; Thirteen-point square mode, wherein a total of 13 image blocks are selected as the plurality of image blocks from the intersections of the five vertical axes and the five horizontal axes of the digital image.

4. The computer-implemented method of claim 1, wherein: Determining the 1D energy spectrum of each processed image block in the plurality of processed image blocks comprises: between a plurality of pixels in each processed image block, The determined 2D energy spectra are accumulated along multiple directions.

5. The computer-implemented method of claim 1, wherein: Determining a 1D energy spectrum of each processed image block in the plurality of processed image blocks comprises: vertically folding the 2D energy spectrum of the processed image block along the x-axis of a coordinate system; The 2D energy values ​​of a plurality of pixels symmetric about the x-axis are summed to obtain a 2D vertically folded energy spectrum.

6. The computer-implemented method of claim 5, wherein: Determining the 1D energy spectrum of each processed image block in the plurality of processed image blocks further comprises: horizontally folding the 2D vertically folded energy spectrum of the processed image block along the y-axis of the coordinate system; summing 2D energy values ​​of a plurality of pixels symmetrical about the y-axis to obtain a 2D horizontally folded energy spectrum; The 1D energy spectrum of the processed image block is determined by accumulating the 2D horizontally folded energy spectra along multiple directions between multiple pixels within the processed image block of the multiple processed image blocks.

7. The computer-implemented method of claim 1, wherein: The adjusting the video characteristics of the digital image comprises: A video stream including the digital image is downsampled according to the maximum image resolution among the plurality of image resolutions determined for the plurality of processed image blocks.

8. The computer-implemented method of claim 1 , wherein: For each processed image block of the plurality of processed image blocks, determining a 2D energy spectrum, determining a 1D energy spectrum, and applying a threshold to the determined 1D energy spectrum to obtain an image resolution are performed in parallel.

9. A computer-implemented system for image resolution assessment of digital images, characterized in that The system comprises: a memory for storing instructions; one or more processors in communication with the memory, wherein the one or more processors execute the instructions to: extracting a plurality of image blocks from a digital image according to a predefined sampling pattern; applying a Gaussian window to each image block of the plurality of image blocks to generate a plurality of processed image blocks; Determining a two-dimensional (2D) energy spectrum of each processed image block in the plurality of processed image blocks by discrete Fourier transformation (DFT); Determine a one-dimensional (1D) energy spectrum of each processed image block in the plurality of processed image blocks using the determined 2D energy spectrum; obtaining an image resolution of each of the plurality of processed image blocks by applying a threshold to the determined 1D energy spectrum; Video characteristics of the digital image are adjusted based on a maximum image resolution among a plurality of image resolutions determined for the plurality of processed image blocks.

10. The system according to claim 9, characterized in that The one or more processors execute the instructions to: The predefined sampling pattern is selected from a plurality of predefined sampling patterns according to the received input.

11. The system according to claim 9, characterized in that Determining the 1D energy spectrum of each processed image block of the plurality of processed image blocks comprises: accumulating the determined 2D energy spectrum along a plurality of directions between a plurality of pixels in each processed image block.

12. The system according to claim 9, characterized in that To determine the 1D energy spectrum of each processed image block of the plurality of processed image blocks, the one or more processors execute the instructions to: vertically folding the 2D energy spectrum of the processed image block along the x-axis of a coordinate system; The 2D energy values ​​of a plurality of pixels symmetric about the x-axis are summed to obtain a 2D vertically folded energy spectrum.

13. The system according to claim 12, characterized in that To determine the 1D energy spectrum of each processed image block of the plurality of processed image blocks, the one or more processors further execute the instructions to: horizontally folding the 2D vertically folded energy spectrum of the processed image block along the y-axis of the coordinate system; summing 2D energy values ​​of a plurality of pixels symmetrical about the y-axis to obtain a 2D horizontally folded energy spectrum; The 1D energy spectrum of the processed image block is determined by accumulating the 2D horizontally folded energy spectra along multiple directions between multiple pixels within the processed image block of the multiple processed image blocks.

14. The system according to claim 9, characterized in that in order to adjust the video characteristics of the digital image, The one or more processors execute the instructions to: A video stream including the digital image is downsampled according to the maximum image resolution among the plurality of image resolutions determined for the plurality of processed image blocks.

15. The system according to claim 9, characterized in that The one or more processors execute the instructions to: For each processed image block of the plurality of processed image blocks, the 2D energy spectrum, the 1D energy spectrum and the image resolution are determined in parallel.

16. A computer readable medium storing computer instructions for image resolution assessment of a digital image, characterized in that When one or more processors execute the instructions, the instructions cause the one or more processors to perform the following operations: extracting a plurality of image blocks from the digital image according to a predefined sampling pattern; applying a Gaussian window to each image block of the plurality of image blocks to generate a plurality of processed image blocks; Determining a two-dimensional (2D) energy spectrum of each processed image block in the plurality of processed image blocks by discrete Fourier transformation (DFT); Determine a one-dimensional (1D) energy spectrum of each processed image block in the plurality of processed image blocks using the determined 2D energy spectrum; obtaining an image resolution of each of the plurality of processed image blocks by applying a threshold to the determined 1D energy spectrum; Video characteristics of the digital image are adjusted based on a maximum image resolution among a plurality of image resolutions determined for the plurality of processed image blocks.

17. The computer-readable medium of claim 16, wherein: The instructions further cause the one or more processors to perform the following steps: The predefined sampling pattern is selected from a plurality of predefined sampling patterns according to the received input.

18. The computer-readable medium of claim 16, wherein: To determine the 1D energy spectrum of each processed image block in the plurality of processed image blocks, the instructions further cause the one or more processors to perform the following steps: The determined 2D energy spectra are accumulated along multiple directions between multiple pixels within each processed image block.

19. The computer-readable medium of claim 16, wherein: To determine the 1D energy spectrum of each processed image block in the plurality of processed image blocks, the instructions further cause the one or more processors to perform the following steps: vertically folding the 2D energy spectrum of the processed image block along the x-axis of a coordinate system; The 2D energy values ​​of a plurality of pixels symmetric about the x-axis are summed to obtain a 2D vertically folded energy spectrum.

20. The computer-readable medium of claim 19, wherein: To determine the 1D energy spectrum of each processed image block in the plurality of processed image blocks, the instructions further cause the one or more processors to perform the following steps: horizontally folding the 2D vertically folded energy spectrum of the processed image block along the y-axis of the coordinate system; summing 2D energy values ​​of a plurality of pixels symmetrical about the y-axis to obtain a 2D horizontally folded energy spectrum; The 1D energy spectrum of the processed image block is determined by accumulating the 2D horizontally folded energy spectra along multiple directions between multiple pixels within the processed image block of the multiple processed image blocks.

Images