Electronic device and its control method

Abstract

An electronic device and a control method thereof are provided. The electronic device includes: a memory configured to store a plurality of filters each corresponding to a plurality of image patterns; and a processor configured to classify an image block into one of the plurality of image patterns based on a relationship between pixels within the image block including a target pixel and a plurality of surrounding pixels, and obtain a final image block by applying at least one filter corresponding to the classified image pattern among the plurality of filters to the image block, wherein the target pixel undergoes image processing in the final image block, and the plurality of filters are obtained by: based on each of the plurality of image patterns, learning a relationship between a plurality of first sample image blocks and a plurality of second sample image blocks corresponding to the plurality of first sample image blocks through an artificial intelligence algorithm.

Description

[0001] Cross - reference to related applications

[0002] This application claims priority to Korean Patent Application No. 10 - 2018 - 0027790, filed on March 9, 2018 with the Korean Intellectual Property Office, and U.S. Provisional Patent Application No. 62 / 599,936, filed on December 18, 2017 with the United States Patent and Trademark Office, and the disclosures of the above - mentioned patent applications are incorporated herein by reference in their entireties. Technical field

[0003] The present disclosure generally relates to an electronic device and a control method thereof, and for example, to an electronic device that performs image processing and a control method thereof.

[0004] The present disclosure also relates to an artificial intelligence (AI) system that uses a machine learning algorithm to simulate the recognition and decision - making functions of the human brain and its applications. Background art

[0005] In recent years, artificial intelligence systems that achieve human - level intelligence have been used in various fields. An artificial intelligence system is a system in which a machine performs learning and decision - making and becomes intelligent itself, different from existing rule - based intelligent systems. As the use of artificial intelligence systems increases, the recognition rate is improved and the user's taste is understood more accurately, making existing rule - based intelligent systems gradually replaced by deep - learning - based artificial intelligence systems.

[0006] Artificial intelligence technology may include machine learning (e.g., deep learning) and meta - technologies that use machine learning.

[0007] Machine learning may include algorithmic techniques that classify / learn the features of input data by itself, and meta - technologies may include techniques that use machine learning algorithms such as deep learning to simulate the functions of the human brain (such as recognition, decision - making, etc.), and may include technical fields such as language understanding, visual understanding, reasoning / prediction, knowledge representation, motion control, etc.

[0008] The various fields to which artificial intelligence technology can be applied are as follows. Language understanding can refer to the technology of recognizing and applying / processing human languages, and can include natural language processing, machine translation, dialogue systems, question and answer, speech recognition / synthesis, etc. Visual understanding can refer to the technology of recognizing and processing things such as human vision, and can include object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, etc. Reasoning / prediction can refer to the technology of judgment and logical reasoning as well as predicting information, and can include knowledge / probability-based reasoning, optimized prediction, preference-based planning, recommendation, etc. Knowledge representation can refer to the technology of automating human experience information and processing it into knowledge data, and can include knowledge construction (data creation / categorization), knowledge management (data utilization), etc. Motion control can refer to the technology of controlling the autonomous driving of vehicles and the motion of robots, and can include motion control (navigation, collision, driving), manipulation control (behavior control), etc.

[0009] At the same time, conventional image processing methods can be divided into non-learning-based technologies and learning-based technologies. Non-learning-based technologies have the advantage of fast image processing speed, but there is a problem that flexible image processing cannot be performed according to image characteristics. Learning-based technologies have the advantage of being able to perform flexible image processing, but there is a problem that it is difficult to process in real time.

[0010] For example, when considering the case of magnifying the resolution of an image, in the interpolation method, which is a representative method of non-learning-based technologies, a filter with low-pass characteristics is used to calculate the brightness of the pixel corresponding to the position where the resolution is magnified. Specifically, there are bicubic interpolation methods based on splines, resampling methods using Lanczos filters formed by simplifying the ideal low-pass filter (Sinc Kernel), etc. This non-learning-based technology shows stable image magnification performance due to its low complexity, but it may not be able to reflect the prior information possessed only by the image, resulting in blurred edge sharpness, jagged edges, aliasing or ringing near the edges.

[0011] As representative methods of learning-based technologies, there are methods of directly using a high-image-quality image database for reconstruction, methods of learning and using high-resolution conversion rules for each classification level, and methods of learning low-resolution / high-resolution conversion in an end-to-end mapping form through a deep learning network and using the learned network to magnify the image when magnifying the image.

[0012] In learning-based techniques, the unique characteristics of an image signal are reflected in learning and used when magnifying an image, and thus, compared with non-learning-based image magnification methods, learning-based techniques can reconstruct sharp, non-aliased, and smooth edges. However, due to high complexity, learning-based techniques are suitable for applications that do not require real-time processing, but it is difficult to apply learning-based techniques to devices that require real-time processing, such as a television (TV). Additionally, it is difficult to apply learning-based techniques to a system-on-chip (SoC) embodiment for real-time implementation.

[0013] In addition, learning-based techniques exhibit excellent performance for edge components with clear regional features, but show noise components for flat regions with unclear features or low performance on a surface with detailed characterization. Additionally, there is a problem that an image may not be magnified to a non-learning magnification factor in learning-based techniques.

[0014] Therefore, there is a need to develop a technique that can perform flexible image processing and improve the image processing speed. SUMMARY OF THE INVENTION

[0015] Exemplary embodiments of the present disclosure solve the above disadvantages and other disadvantages not described above.

[0016] The present disclosure provides an electronic device for performing learning-based image processing in real time and a control method thereof.

[0017] According to an exemplary aspect of the present disclosure, an electronic device includes: a memory configured to store a plurality of filters, each filter corresponding to a plurality of image patterns; and a processor configured to classify an image block into one of the plurality of image patterns based on a relationship between pixels within the image block including a target pixel and a plurality of surrounding pixels, and obtain a final image block by applying at least one filter corresponding to the classified image pattern among the plurality of filters to the image block, wherein the target pixel in the final image block has undergone image processing, and the plurality of filters are obtained by: based on each of the plurality of image patterns, learning a relationship between a plurality of first sample image blocks and a plurality of second sample image blocks corresponding to the plurality of first sample image blocks through an artificial intelligence algorithm.

[0018] The processor may calculate (determine) a gradient vector from the image block, calculate (determine) a correlation between the image block and each of the plurality of index vectors based on the gradient vector and an index matrix stored in the memory and including the plurality of index vectors, and classify the image block into one of the plurality of image patterns based on the calculated (determined) correlation, and the plurality of index vectors may be obtained based on a plurality of sample gradient vectors calculated (determined) from the plurality of first sample image blocks and corresponding to the plurality of image patterns, respectively.

[0019] When the intensity of the horizontal directionality of an image block is greater than the intensity of the vertical directionality of the image block, the processor may calculate a gradient vector from the image block, and when the intensity of the horizontal directionality of the image block is less than the intensity of the vertical directionality of the image block, the processor may transpose the image block and calculate a gradient vector from the transposed image block.

[0020] The processor may use eigenvectors stored in a memory to reduce the size of the gradient vector, and calculate the correlation between the image block and each of a plurality of index vectors based on the gradient vector with reduced size, and the eigenvectors may be created by applying principal component analysis to a plurality of sample gradient vectors.

[0021] The processor may calculate a first correlation between the image block and each of a plurality of index vector groups based on the gradient vector and a first index matrix stored in the memory, obtain one of the plurality of index vector groups based on the first correlation, calculate a second correlation between the image block and each of the plurality of index vectors included in the obtained index vector group based on the gradient vector and a second index matrix corresponding to the obtained index vector group among a plurality of second index matrices stored in the memory, and obtain a final image block by applying at least one of a plurality of filters to the image block based on the second correlation. The plurality of index vector groups may be obtained by dividing the plurality of index vectors into a predetermined number of groups. The first index matrix may include index vectors representing each of the plurality of index vector groups, and each of the plurality of second index matrices may include a plurality of index vectors corresponding to each of the plurality of index vector groups.

[0022] The processor may obtain one of the plurality of index vectors included in the obtained index vector group based on the second correlation, obtain at least one additional index vector included in other index vector groups among the plurality of index vector groups and corresponding to the obtained index vector based on similar index vector information stored in the memory, calculate a third correlation of the image block based on the gradient vector, the obtained index vector, and the obtained additional index vector, and obtain a final image block by applying at least one of a plurality of filters to the image block based on the third correlation.

[0023] The processor may obtain at least two of the plurality of filters based on the calculated plurality of correlations, calculate a final filter based on the at least two filters and the correlations of the image block corresponding to each of the at least two filters, and obtain a final image block by applying the calculated final filter to the image block.

[0024] Multiple first sample image patches may be images in which the corresponding second sample image patches have been respectively downsampled, and the processor may obtain a final image patch with an enlarged resolution of a target pixel by applying at least one filter from multiple filters to the image patch.

[0025] The processor may obtain an additional image patch with an enlarged resolution of a target pixel by applying a non-learning-based resolution magnification technique to the image patch, and update the final image patch based on the maximum correlation among multiple computed correlations and the additional image patch.

[0026] The processor may compute a horizontal gradient and a vertical gradient of each of multiple surrounding pixels and the target pixel, and compute a gradient vector based on the horizontal gradient and the vertical gradient.

[0027] Each of the multiple filters may be obtained by: obtaining multiple first subsample image patches corresponding to one index vector from multiple index vectors from among multiple first sample image patches, obtaining multiple second subsample image patches corresponding to the multiple first subsample image patches from among multiple second sample image patches, and learning the relationship between the multiple first subsample image patches and the multiple second subsample image patches through an artificial intelligence algorithm.

[0028] According to another exemplary aspect of the present disclosure, a method of controlling an electronic device includes: classifying an image patch into one of multiple image patterns based on a relationship between pixels within the image patch including a target pixel and multiple surrounding pixels; and obtaining a final image patch by applying at least one filter corresponding to the classified image pattern from among multiple filters each corresponding to multiple image patterns to the image patch, in which the target pixel in the final image patch has undergone image processing, where the multiple filters may be obtained by: learning the relationship between multiple first sample image patches and multiple second sample image patches corresponding to the multiple first sample image patches through an artificial intelligence algorithm based on each of the multiple image patterns.

[0029] The classification may include: computing (determining) a gradient vector from the image patch; computing (determining) a correlation between the image patch and each of multiple index vectors based on an index matrix including multiple index vectors and the gradient vector; and classifying the image patch into one of the multiple image patterns based on the computed (determined) correlations, and the multiple index vectors may be obtained based on multiple sample gradient vectors computed from multiple first sample image patches, and the multiple index vectors may respectively correspond to the multiple image patterns.

[0030] In the process of calculating the gradient vector: when the intensity of the horizontal directionality of the image patch is greater than the intensity of the vertical directionality of the image patch, the gradient vector can be calculated from the image patch; and when the intensity of the horizontal directionality of the image patch is less than the intensity of the vertical directionality of the image patch, the image patch can be transposed and the gradient vector can be calculated from the transposed image patch.

[0031] The method may further include reducing the size of the gradient vector using eigenvectors, wherein, in the process of calculating the correlation, the correlation between the image patch and each index vector in the plurality of index vectors is calculated based on the gradient vector with reduced size, and the eigenvectors are created by applying principal component analysis of the plurality of sample gradient vectors.

[0032] Calculating the correlation may include: calculating a first correlation between the image patch and each group of index vectors in the plurality of groups of index vectors based on the first index matrix and the gradient vector; obtaining one group of index vectors from the plurality of groups of index vectors based on the first correlation; calculating a second correlation between the image patch and each index vector included in the obtained group of index vectors based on the gradient vector and the second index matrix corresponding to the obtained group of index vectors among the plurality of second index matrices; and in the process of obtaining the final image patch, the final image patch can be obtained by applying at least one filter from the plurality of filters to the image patch based on the second correlation. The plurality of groups of index vectors can be obtained by dividing the plurality of index vectors into a predetermined number of groups. The first index matrix may include index vectors representing each group of index vectors in the plurality of groups of index vectors, and each second index matrix among the plurality of second index matrices may include a plurality of index vectors corresponding to each group of index vectors in the plurality of groups of index vectors.

[0033] Calculating the correlation may include: obtaining one index vector from the plurality of index vectors included in the obtained group of index vectors based on the second correlation; obtaining at least one additional index vector included in other groups of index vectors in the plurality of groups of index vectors and corresponding to the obtained index vector based on similar index vector information; and calculating a third correlation between the image patch, the obtained index vector, and the obtained additional index vector based on the gradient vector. And in the process of obtaining the final image patch, the final image patch can be obtained by applying at least one filter from the plurality of filters to the image patch based on the third correlation.

[0034] Obtaining the final image patch may include: obtaining at least two filters from the plurality of filters based on the calculated plurality of correlations; calculating a final filter based on the at least two filters and the correlations corresponding to each of the at least two filters and the image patch; and obtaining the final image patch by applying the calculated final filter to the image patch.

[0035] Multiple first sample image blocks may be images in which the corresponding second sample image blocks have been respectively downsampled, and in the process of obtaining the final image block, the final image block with the resolution of the target pixel enlarged may be obtained by applying at least one filter among multiple filters to the image block.

[0036] The method may further include: obtaining an additional image block with the resolution of the target pixel enlarged by applying a non-learning-based resolution magnification technique to the image block; and updating the final image block based on the maximum correlation among the calculated multiple correlations and the additional image block.

[0037] In the process of calculating the gradient vector, the horizontal gradient and the vertical gradient of each of the multiple surrounding pixels and the target pixel may be calculated, and the gradient vector may be calculated based on the horizontal gradient and the vertical gradient.

[0038] Each of the multiple filters may be obtained by: obtaining multiple first subsample image blocks corresponding to one index vector among multiple index vectors from among the multiple first sample image blocks, obtaining multiple second subsample image blocks corresponding to the multiple first subsample image blocks from among the multiple second sample image blocks, and learning the relationship between the multiple first subsample image blocks and the multiple second subsample image blocks through an artificial intelligence algorithm.

[0039] According to various exemplary embodiments of the present disclosure as described above, the electronic device may perform image processing in real time and thus be used in various applications, and the electronic device may improve the quality of image processing compared to non-learning-based image processing by performing learning-based image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] From the following detailed description in conjunction with the drawings, the above and / or other aspects, features, and accompanying beneficial effects of the present disclosure will become more obvious and easier to understand. In the drawings, the same reference numerals denote the same elements, and in which:

[0041] Figure 1A is a block diagram showing components of an exemplary electronic device according to an exemplary embodiment of the present disclosure;

[0042] Figure 1B is a block diagram showing an example of components of an electronic device;

[0043] Figure 2 is a diagram showing an exemplary operation of a processor according to an exemplary embodiment of the present disclosure;

[0044] Figure 3A 、 Figure 3B 、 Figure 3C 、 Figure 3D and Figure 3Eis a diagram illustrating an exemplary method of determining the orientation of an image block and creating a gradient vector according to an exemplary embodiment of the present disclosure;

[0045] Figure 4A and Figure 4B is a diagram illustrating an exemplary dimensionality reduction of the gradient vector according to an exemplary embodiment of the present disclosure;

[0046] Figure 5A and Figure 5B is a diagram illustrating an exemplary search of a filter using an index matrix according to an exemplary embodiment of the present disclosure;

[0047] Figure 6A 、 Figure 6B and Figure 6C is a diagram illustrating an exemplary method of reducing the search computational amount of a filter according to an exemplary embodiment of the present disclosure;

[0048] Figure 7A and Figure 7B is a diagram illustrating an exemplary method of reducing the search computational amount of a filter according to another exemplary embodiment of the present disclosure;

[0049] Figure 8A and Figure 8B is a diagram illustrating an exemplary method of applying a filter to an image block according to an exemplary embodiment of the present disclosure;

[0050] Figure 9A and Figure 9B is a diagram illustrating an exemplary learning method of a filter according to an exemplary embodiment of the present disclosure;

[0051] Figure 10 is a flowchart illustrating an exemplary stream operation according to an exemplary embodiment of the present disclosure; and

[0052] Figure 11 is a flowchart illustrating an exemplary method of controlling an electronic device according to an exemplary embodiment of the present disclosure. Detailed Description

[0053] The various exemplary embodiments of the present disclosure can be modified differently. Therefore, specific exemplary embodiments are shown in the drawings and are described in detail in the present disclosure. However, it will be understood that the present disclosure is not limited to the specific exemplary embodiments, but includes all modifications, equivalents, and alternatives without departing from the scope and spirit of the present disclosure. In addition, well-known functions or structures may not be described in detail, as they would obscure the present disclosure with unnecessary details.

[0054] Hereinafter, various exemplary embodiments of the present disclosure will be described in more detail with reference to the drawings.

[0055] Figure 1A is a block diagram showing exemplary components of an electronic device 100 according to an exemplary embodiment of the present disclosure. As Figure 1A shown, the electronic device 100 includes a memory 110 and a processor (e.g., including processing circuitry) 120.

[0056] The electronic device 100 may be a device that performs image processing. For example, the electronic device 100 may be a device that performs image processing based on data learned through an artificial intelligence algorithm. For example, the electronic device 100 may be, for example but not limited to, a desktop personal computer (PC), a laptop computer, a smartphone, a tablet PC, a server, a refrigerator, a washing machine, a signboard, etc. Additionally, the electronic device 100 may be, for example but not limited to, a system that is itself configured with a cloud computing environment. However, the electronic device 100 is not limited thereto, and may be any device that can perform image processing.

[0057] The memory 110 may store a plurality of filters corresponding to each of a plurality of image patterns. Here, the plurality of image patterns may be classified according to image characteristics. For example, the first image pattern may be an image pattern having many lines in the horizontal direction, and the second image pattern may be an image pattern having many lines in the rotational direction.

[0058] The memory 110 may also store an index matrix including a plurality of index vectors. The plurality of index vectors may be obtained based on a plurality of sample gradient vectors calculated (determined) from a plurality of first sample image blocks, and the plurality of index vectors may respectively correspond to the plurality of image patterns. The plurality of filters may be obtained by: based on each of the plurality of index vectors, learning the relationship between the plurality of first sample image blocks and the plurality of second sample image blocks corresponding to the plurality of first sample image blocks through an artificial intelligence algorithm.

[0059] The plurality of index vectors and the plurality of filters may be created by an external server rather than the electronic device 100. However, the plurality of index vectors and the plurality of filters are not limited thereto, and may also be created by the electronic device 100. In this case, the memory 110 may store the plurality of first sample image blocks and the plurality of second sample image blocks.

[0060] A method of creating gradient vectors, the plurality of index vectors, and the plurality of filters will be described below.

[0061] The memory 110 may store at least one image block. The memory 110 may store a plurality of first sample image blocks and a plurality of second sample image blocks. Additionally, the memory 110 may store filters not based on learning, and the type of the memory is not limited.

[0062] The memory 110 can be implemented, for example but not limited to, by a hard disk, non-volatile memory, volatile memory, etc., and can be any component that can store data.

[0063] The processor 120 can include various processing circuits and generally control the operation of the electronic device 100.

[0064] According to an exemplary embodiment, the processor 120 can be implemented by, for example but not limited to, a digital signal processor (DSP), a microprocessor, and / or a timing controller (TCON), etc. However, the processor 120 is not limited thereto and can include, for example but not limited to, one or more of the following: a dedicated processor, a central processing unit (CPU), a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an ARM processor, and / or can be defined by these terms, etc. Additionally, the processor 120 can be implemented by a system-on-chip (SoC) or a large-scale integrated circuit (LSI) embedded with a processing algorithm, and / or can be implemented in the form of a field-programmable gate array (FPGA).

[0065] The processor 120 can receive an image block including a target pixel and a plurality of surrounding pixels input from an external server. In this case, the processor 120 can store the image block in the memory 110. The processor 120 can read the image block stored in the memory 110. The processor 120 can read an image block having a predetermined size from an image frame stored in the memory 110. For example, the processor 120 can read a 3×3 first image block at the upper left end of the image frame and perform image processing on the first image block. Additionally, the processor 120 can read a 3×3 second image block that is shifted one unit pixel to the right from the upper left end of the image frame and perform image processing on the second image block. In this way, the processor 120 can perform image processing on the entire image frame.

[0066] The target pixel can be a pixel that becomes a filtering target described below. That is, the target pixel can be filtered based on the pixel values of a plurality of surrounding pixels.

[0067] The processor 120 can classify an image block into one of a plurality of image patterns based on the relationship between pixels within the image block including the target pixel and a plurality of surrounding pixels. For example, when the difference between the pixel values of pixels adjacent to each other in the horizontal direction is less than the difference between the pixel values of pixels adjacent to each other in the vertical direction, the processor 120 can classify the image block into an image pattern having a vertical direction characteristic.

[0068] However, the present disclosure is not limited thereto, and the processor 120 can also rely on various methods to obtain the relationship between pixels within the image block.

[0069] In addition, the processor 120 can obtain a final image block by applying at least one filter corresponding to the classified image pattern among the multiple filters to the image block, and image processing is performed on the target pixel in the final image block.

[0070] As another example for classifying an image block, the processor 120 can calculate (determine) a gradient vector from the image block. For example, the processor 120 can calculate the horizontal gradient and the vertical gradient for each of the multiple surrounding pixels and the target pixel, and calculate the gradient vector based on the horizontal gradient and the vertical gradient. The gradient vector can indicate the amount of change of the pixels existing in a predetermined direction relative to each pixel. That is, the characteristics of the image block can be detected through the gradient vector.

[0071] For example, the processor 120 can calculate the horizontal gradient and the vertical gradient of a 3×3 image block using the Sobel operator. Each of the horizontal gradient and the vertical gradient can include nine elements, and the processor 120 can arrange the elements of each of the horizontal gradient and the vertical gradient in a predetermined order and convert the elements into a vector form. The predetermined order can be set by various methods. However, the predetermined order can be the same as the order for calculating the gradient vector in the process of creating the multiple index vectors and the multiple filters described below.

[0072] In addition to the Sobel operator, the processor 120 can also calculate the gradient vector of the image block by various methods, and the various methods are not particularly limited. In addition to the horizontal gradient vector and the vertical gradient vector, a gradient vector with another angle can also be calculated, and any method can be used as long as the gradient vector is calculated by the same method as the method used in the process of creating the multiple index vectors and the multiple filters described below. Hereinafter, for the sake of convenience of explanation, the case of calculating a c×1 gradient vector is described. Here, c can vary according to the size of the image block.

[0073] The processor 120 can calculate the correlation between the image block and each of the multiple index vectors based on the gradient vector and the index matrix stored in the memory 110. The index matrix can include multiple index vectors, and the multiple index vectors can be obtained based on the multiple sample gradient vectors calculated from the multiple first sample image blocks.

[0074] For example, in the case where 1,000,000 sample gradient vectors calculated from 1,000,000 first sample image patches are divided into groups for each feature in the features of the image patches, each index vector among the multiple index vectors can be a vector representing each group. That is, the correlation between an image patch and each index vector among the multiple index vectors can mean the similarity between the image patch and each of the a groups. Additionally, the image patch can be considered to have features similar to those of the group corresponding to the maximum correlation among the a correlations.

[0075] However, this is only an example, and the number of first sample image patches and the number of groups can be changed. Additionally, the number of groups can also be changed according to the type of filter. For example, in the case of a resolution magnification filter, the number of groups is a, but in the case of a sharpening filter, the number of groups can be different from a.

[0076] The processor 120 can obtain a final image patch by applying at least one filter among the multiple filters to the image patch based on the calculated correlation, in which the target pixels are processed. Here, the filter can be a resolution magnification filter, a sharpening filter, etc., but is not limited thereto, and can also be any filter related to image processing.

[0077] Additionally, the multiple filters can be obtained by the following method: based on each index vector among the multiple index vectors, learning the relationship between the multiple first sample image patches and the multiple second sample image patches corresponding to the multiple first sample image patches through an artificial intelligence algorithm.

[0078] For example, each filter among the multiple filters can be obtained by the following method: obtaining multiple first sub-sample image patches corresponding to one index vector among the multiple index vectors from among the multiple first sample image patches, obtaining multiple second sub-sample image patches corresponding to the multiple first sub-sample image patches from among the multiple second sample image patches, and learning the relationship between the multiple first sub-sample image patches and the multiple second sub-sample image patches through an artificial intelligence algorithm.

[0079] In the above example, each of the multiple filters can be, for example but not limited to, a resolution magnification filter obtained by learning the image characteristics of one of a groups through an artificial intelligence algorithm. When the first image characteristic is represented by a first index vector and there is a first filter corresponding to the first index vector, and the second image characteristic is represented by a second index vector and there is a second filter corresponding to the second index vector, when the correlation between the image patch and the first image characteristic is high, the image patch can show higher quality when its resolution is magnified by the first filter compared to when its resolution is magnified by the second filter. The reason is that the first filter and the second filter are filters learned to be respectively suitable for the first image characteristic and the second image characteristic.

[0080] When the intensity of the horizontal directionality of the image patch is greater than the intensity of the vertical directionality of the image patch, the processor 120 can calculate the gradient vector from the image patch, and when the intensity of the horizontal directionality of the image patch is less than the intensity of the vertical directionality of the image patch, the processor 120 can transpose the image patch and calculate the gradient vector from the transposed image patch.

[0081] For example, the processor 120 can transpose an image patch with dominant vertical directionality into an image patch with dominant horizontal directionality, and calculate the gradient vector from the image patch with dominant horizontal directionality. In this case, the number of index vectors can be reduced, thereby improving the calculation speed.

[0082] For example, in the case of dividing the image patches with horizontal directionality into a first a groups for each image characteristic, the image patches with vertical directionality can also be divided into a second a groups for each image characteristic. Therefore, for each image characteristic, a total of 2×a groups can be formed, and a total of 2×a index vectors can be created.

[0083] Therefore, when the processor 120 transposes an image patch with dominant vertical directionality, the transposed image patch can have horizontal directionality and can be included in one of the first groups. That is, the processor 120 can process the image patch belonging to one of the second groups into one of the first groups by transposing the image patch with dominant vertical directionality, and can process the image patch with dominant horizontal directionality in one of the first groups without transposing the image patch with dominant horizontal directionality. Therefore, according to the transposition operation, the number of index vectors can be reduced by half, and the calculation speed can be improved.

[0084] The processor 120 can obtain the final image block by applying a filter to the transposed image block and then transposing the image block again. In the final image block, the target pixels have undergone image processing. When the image block is not transposed, the operations of applying the filter to the image block and then transposing the image block again can be omitted.

[0085] Meanwhile, although the case where the processor 120 transposes the image block with dominant vertical directionality has been described above, this is only an example, and the processor 120 can also transpose the image block with dominant horizontal directionality.

[0086] Meanwhile, the processor 120 can use the eigenvectors stored in the memory 110 to reduce the size of the gradient vector and calculate the correlation between the image block and each of the multiple index vectors based on the gradient vector with reduced size. Here, the eigenvectors can be created by applying principal component analysis of multiple sample gradient vectors.

[0087] When the eigenvectors are not used in the state where the index vectors exist, the index matrix can have the form of a×c. In this regard, when using the eigenvectors to reduce the c×1 gradient vector to a c'×1 gradient vector, the index matrix can have the form of a×c'. Specifically, the characteristic sizes of the multiple index vectors included in the a×c index matrix can be reduced using the same eigenvectors, and the index matrix including the multiple index vectors with reduced characteristic sizes can have the form of a×c’. Here, the eigenvectors can have the form of c’×c, and c’ can be smaller than c.

[0088] Compared with the case of using the a×c index matrix, the calculation speed can be improved in the case of using the a×c’ index matrix.

[0089] Meanwhile, principal component analysis is a statistical method for extracting principal components, which concisely represents the variation type patterns of many variables as a linear combination of the original variables. That is, in the presence of p variables, the information obtained from the p variables can be reduced to k variables significantly smaller than p. In the above example, it can be considered that the c characteristic size is reduced to the c' characteristic size. However, the number of reduced characteristic sizes is not limited and can also be changed.

[0090] Hereinafter, the case of using eigenvectors to reduce the index matrix and the gradient vector has been described. However, the present disclosure is not limited thereto, and the configuration without using eigenvectors corresponds to the exemplary embodiments of the present disclosure.

[0091] Meanwhile, the processor 120 may calculate a first correlation between the image block and each of a plurality of index vector groups based on the gradient vector and a first index matrix stored in the memory 110, obtain one of the plurality of index vector groups based on the first correlation, and obtain a second index matrix corresponding to the obtained index vector from among a plurality of second index matrices stored in the memory 110.

[0092] Here, the plurality of index vector groups may be obtained by dividing the plurality of index vectors into a predetermined number of groups. For example, the plurality of index vector groups may be divided into a total of four groups according to characteristics, and each of the four groups may include b index vectors. The b index vectors in the same group may be similar to each other.

[0093] The first index matrix may include index vectors representing each of the plurality of index vector groups. In the above example, in each of the four groups, one of the b index vectors may be set as a representative, and the four index vectors representing the respective groups may be used to form the first index matrix. Here, the first index matrix may have a form of 4×c’.

[0094] Each of the plurality of second index matrices may include a plurality of index vectors corresponding to each of the plurality of index vector groups. For example, each of the plurality of index vectors may have a form of b×c’.

[0095] In the above example, the processor 120 may determine which index vector group the image block belongs to among the plurality of index vector groups based on the first correlation, and obtain the second index matrix corresponding to the corresponding group from among the plurality of second index matrices.

[0096] The processor 120 may calculate a second correlation between the image block and each of the plurality of index vectors included in the obtained index vectors based on the gradient vector and the obtained second index matrix, and may obtain a final image block by applying at least one filter among the plurality of filters to the image block based on the second correlation. The calculation speed may be improved by the method described above.

[0097] However, this is only an example, and the number of index vector groups may be changed. In addition, the number of index vectors included in the plurality of index vector groups may be different from each other.

[0098] Meanwhile, according to the above exemplary embodiment, the correlation with only some of a total of a index vectors may be calculated. Therefore, the calculation of the correlation with other index vectors is omitted, and in the case of improper grouping, the index vector with the highest correlation may not be detected. Hereinafter, a method for supplementing this is described.

[0099] The processor 120 may obtain one index vector from among the multiple index vectors included in the obtained index vector group based on the second correlation, obtain at least one additional index vector included in another index vector group among the multiple index vector groups and corresponding to the obtained index vector based on the similar index vector information stored in the memory 110, calculate a third correlation of the image block based on the gradient vector, the obtained index vector, and the obtained additional index vector, and obtain a final image block by applying at least one filter among the multiple filters to the image block based on the third correlation.

[0100] That is, the memory 110 may also store similar index vector information, which is information about additional index vectors included in different index vector groups having similar characteristics for each index vector.

[0101] When obtaining one index vector based on the second correlation, the processor 120 may calculate the correlation between the image block and an additional index vector based on the information stored in the memory 110, where the additional index vector has characteristics similar to those of the obtained index vector. The processor 120 may obtain at least one filter based on the correlation between the image block and the obtained index vector and the correlation between the image block and the additional index vector.

[0102] In this case, additional multiplications are performed during the process of calculating the correlation between the image block and the additional index vector, so that the speed may be reduced compared to the previous exemplary embodiment, but the accuracy may be improved compared to the previous exemplary embodiment. Additionally, when the number of additional index vectors is not very large, the reduction in speed may not be significant.

[0103] The processor 120 may obtain at least two filters among the multiple filters based on the calculated multiple correlations, calculate a final filter based on the at least two filters and the correlations corresponding to each of the at least two filters, and obtain a final image block by applying the calculated final filter to the image block.

[0104] For example, the processor 120 may obtain a first filter and a second filter, and calculate a final filter by performing a weighted sum of the first filter and the second filter based on a first correlation and a second correlation corresponding to each of the first filter and the second filter.

[0105] However, the present disclosure is not limited thereto, and the processor 120 may also calculate a final filter from the first filter and the second filter without considering the correlations.

[0106] Meanwhile, multiple first sample image blocks can be images with the resolutions of corresponding second sample image blocks reduced respectively, and the processor 120 can obtain a final image block with the resolution of target pixels enlarged by applying at least one filter among multiple filters to the image blocks.

[0107] In this case, the multiple filters can be filters learned and obtained through an artificial intelligence algorithm to derive a corresponding second sample image block with the resolution enlarged from each first sample image block among the multiple first sample image blocks.

[0108] The processor 120 can obtain an additional image block with the resolution of target pixels enlarged by applying a non-learning-based resolution magnification technique to the image blocks, and update the final image block based on the maximum correlation among the calculated multiple correlations and the additional image block.

[0109] For example, the processor 120 can select the maximum correlation among the calculated multiple correlations and determine the change level of the final image block based on the selected correlation. Specifically, the processor 120 can minimize and / or reduce the change of the final image block by reducing the influence of the additional image block when the selected correlation becomes larger, and magnify the change of the final image block by magnifying the influence of the additional image block when the selected correlation becomes smaller.

[0110] Figure 1B is a block diagram showing exemplary components of the electronic device 100. As Figure 1B shown, the electronic device 100 includes a memory 110, a processor (e.g., including a processing circuit) 120, a communicator (e.g., including a communication circuit) 130, a display 140, a user interface 150, an audio processor (e.g., including an audio processing circuit) 160, and a video processor (e.g., including a video processing circuit) 170. For Figure 1B among the components shown in Figure 1A and overlapping with these components shown in

[0111] The processor 120 can include various processing circuits and generally control the operation of the memory 110 using various programs stored in the memory 110.

[0112] For example, the processor 120 can include but is not limited to a random access memory (RAM) 121, a read-only memory (ROM) 122, a main central processing unit (CPU) 123, a graphics processor 124, a first interface 125-1 to an nth interface 125-n, and a bus 126.

[0113] The RAM 121, ROM 122, main CPU 123, graphics processor 124, first interface 125-1 to nth interface 125-n, etc. can be connected to each other via a bus 126.

[0114] The first interface 125-1 to nth interface 125-n are connected to various components as described above. One of the interfaces can be a network interface that is connected to an external server via a network.

[0115] The main CPU 123 accesses the memory 110 to perform startup using the operating system (O / S) stored in the memory 110. In addition, the main CPU 123 performs various operations using various programs stored in the memory 110 and the like.

[0116] The instruction system for starting up the system and the like can be stored in the ROM 122. When a power-on command is input to supply power to the main CPU 123, the main CPU 123 copies the operating system (O / S) stored in the memory 110 to the RAM 121 according to the instructions stored in the ROM 122, and executes the O / S to start up the system. When the startup is completed, the main CPU 123 copies various application programs stored in the memory 110 to the RAM 121, and executes the application programs copied to the RAM 121 to perform various operations.

[0117] The graphics processor 124 uses a calculator (not shown) and a renderer (not shown) to render a screen including various objects such as icons, images, text, etc. The calculator (not shown) calculates attribute values such as the coordinate values at which the corresponding object will be displayed, the shape, size, color, etc. of the corresponding object, based on the received control command and according to the layout of the screen. The renderer (not shown) renders a screen including various layouts of objects based on the attribute values calculated in the calculator (not shown). The screen rendered by the renderer (not shown) is displayed on the display area of the display 140.

[0118] Meanwhile, the operations of the processor 120 as described above can be executed by a program stored in the memory 110.

[0119] The memory 110 stores various data, such as an image block analysis module, a filter module, an operating system (O / S) software module for driving the electronic device 100, etc.

[0120] The communicator 130 may include various communication circuits, and the communicator 130 is a component that performs communication with various types of external servers in various types of communication manners. The communicator 130 may include various communication circuits included in various communication chips, such as but not limited to a Wi-Fi (WiFi) chip 131, a Bluetooth chip 132, a wireless communication chip 133, a near field communication (NFC) chip 134, etc. The processor 120 uses the communicator 130 to perform communication with various external servers.

[0121] The WiFi chip 131 and the Bluetooth chip 132 perform communication in the WiFi manner and the Bluetooth manner, respectively. In the case of using the WiFi chip 131 or the Bluetooth chip 132, various types of connection information such as a service set identifier (SSID), a session key, etc. may be first sent and received, communication may be connected using the connection information, and then various types of information may be sent and received. The wireless communication chip 133 refers to a chip that performs communication according to various communication protocols such as the Institute of Electrical and Electronics Engineers (IEEE), ZigBee, the third generation of mobile communication (3G), the 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE), etc. The NFC chip 134 refers to a chip that operates in the NFC manner using the 13.56 MHz frequency band in various radio frequency identification (RFID) frequency bands such as 135 kHz, 13.56 MHz, 433 MHz, 860 MHz to 960 MHz, 2.45 GHz, etc.

[0122] The processor 120 may receive an index matrix, a plurality of filters, etc. from an external server through the communicator 130. Alternatively, the processor 120 may receive image blocks, etc. from an external server in real time through the communicator 130.

[0123] The display 140 may be implemented by various types of displays, such as but not limited to a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display panel (PDP), etc. The display 140 may include a driving circuit, a backlight unit, etc. that may be implemented in forms such as a-si, low temperature polycrystalline silicon (LTPS), thin film transistor (TFT), organic TFT (OTFT), etc. At the same time, the display 140 may be implemented by a touch screen, in which a touch sensor is incorporated.

[0124] The processor 120 may control the display 140 to display image blocks and final image blocks.

[0125] The user interface 150 receives various user interactions. Here, the user interface 150 may be implemented in various forms according to embodiments of the electronic device 100. For example, the user interface 150 may include various interface circuits, such as but not limited to, a microphone that receives user voice, a camera that senses user movement, buttons included in the electronic device 100, etc. In the case where the electronic device 100 is implemented as a touch-based electronic device, the user interface 150 may also be implemented in the form of a touch screen, which forms a layered structure together with a touch pad. In this case, the user interface 150 may be used as the display 140 described above.

[0126] The audio processor 160 is a component that processes audio data. The audio processor 160 may include various audio processing circuits and may perform various types of processing, such as but not limited to, decoding, amplifying, and noise filtering of audio data.

[0127] The video processor 170 is a component that processes video data. The video processor 170 may include various video processing circuits and may perform various types of image processing, such as but not limited to, decoding, scaling, noise filtering, frame rate conversion, and resolution conversion of video data.

[0128] Meanwhile, the model including an index matrix, multiple filters, etc., provided by an external server as described above is a decision-making model learned based on an artificial intelligence algorithm and may be, for example, a neural network-based model. The learned decision-making model may be designed to simulate the human brain structure on a computer and may include multiple network nodes that simulate neurons of the human neural network and have weights. The multiple network nodes may form connection relationships therein to simulate the synaptic activities of neurons that send and receive signals through synapses. In addition, the learned decision-making model may include, for example, a neural network model or a deep learning model developed from a neural network model. In the deep learning model, the multiple network nodes may be located at different depths (or layers) and may send and receive data therein according to convolutional connection relationships. Examples of the learned decision-making model may include, for example, but not limited to, a deep neural network (DNN), a recurrent neural network (RNN), a bidirectional recurrent deep neural network (BRDNN), etc., but are not limited thereto.

[0129] In addition, the electronic device 100 may use a personal secretary program, which is an artificial intelligence dedicated program (or artificial intelligence agent), to receive the final image block obtained by applying a filter to the image block as described above. In this case, the personal secretary program, which is a dedicated program for providing artificial intelligence (AI)-based services, may be executed by an existing general-purpose processor (e.g., a central processing unit (CPU)) or a separate AI dedicated processor (e.g., a graphics processing unit (GPU)).

[0130] When a predetermined user input is entered (e.g., image capture depending on the manipulation of a capture button, etc.) or an image block is input from an external device, the artificial intelligence agent can be operated (or executed).

[0131] According to another exemplary embodiment of the present disclosure, the electronic device 100 can send the input image block to an external server. The external server can obtain a final image block by directly applying a filter to the image block and send the obtained final image block to the electronic device 100.

[0132] The processor 120 can improve the quality of image processing by performing learning-based image processing in real time through the method described above.

[0133] Hereinafter, the exemplary operations of the processor 120 will be described in more detail with reference to the drawings.

[0134] Figure 2 is a diagram illustrating exemplary operations of the processor 120 according to an exemplary embodiment of the present disclosure.

[0135] When an image block is input, the processor 120 can first perform gradient extraction 210 of the image block. For example, the processor 120 can determine the directionality of the image block and create a gradient vector. In addition, the processor 120 can also transpose the image block based on the determination result of the directionality of the image block.

[0136] The processor 120 can perform feature dimension reduction 220 of the gradient vector using the eigenvector 260.

[0137] The processor 120 can perform filter search 230 on the filter using the index matrix 270, and the filter will be applied to the gradient vector whose feature dimension has been reduced.

[0138] The processor 120 can obtain at least one filter among the multiple filters included in the filter database DB 280 based on the search result and apply the at least one filter to the image block (filter application 240).

[0139] The multiple filters included in the filter database 280 can be learning-based filters. For example, the multiple filters can be obtained by the following method: learning the relationship between multiple first sample image blocks and multiple second sample image blocks corresponding to the multiple first sample image blocks through an artificial intelligence algorithm. That is, the operations described above can be considered as applying learning-based filters to the image block.

[0140] Meanwhile, independently of the operations described above, the processor 120 may apply a non-learning-based filter to the image block (application 250 of the non-learning-based filter). For example, the processor 120 may perform the application of the learning-based filter and the application of the non-learning-based filter to the same image block in parallel.

[0141] In addition, the processor 120 may perform a weighted sum of a first result depending on the application of the learning-based filter and a second result depending on the application of the non-learning-based filter.

[0142] Meanwhile, the eigenvector 260, the index matrix 270, and the filter database 280 may be created using a plurality of first sample image blocks and a plurality of second sample image blocks corresponding to the plurality of first sample image blocks.

[0143] In Figure 2 , the transposition of the image block depending on the directionality determination result, the reduction of the feature dimension of the gradient vector, and the application of the non-learning-based filter are optional operations and may be omitted.

[0144] Figure 3A , Figure 3B , Figure 3C , Figure 3D and Figure 3E are diagrams illustrating an exemplary method of determining the directionality of an image block and creating a gradient vector according to an exemplary embodiment of the present disclosure.

[0145] As shown in FIG. 3, the processor 120 may determine the directionality of the image block using masks such as Sx and Sy. For example, the processor 120 may determine the directionality of the image block by applying a 5×5 image block and a mask to the following Equation 1:

[0146] [Equation 1]

[0147] G V = ∑∑s x (n, m)Y(i + n, j + m)

[0148] G H = ∑∑s y (n, m)Y(i + n, j + m)

[0149] Here, G V indicates the vertical directionality of the image block, G H indicates the horizontal directionality of the image block, n and m respectively indicate indexes for identifying rows and columns, and i and j indicate reference points of the image block in the image frame.

[0150] The processor 120 may obtain the directionality of the image block by comparing the magnitude of the vertical directionality with the magnitude of the horizontal directionality.

[0151] The processor 120 may determine whether to transpose an image block based on the directionality used when creating the eigenvector, index matrix, and multiple filters. For example, when only horizontal directionality is used and vertical directionality is not used when creating the eigenvector, index matrix, and multiple filters, the processor 120 does not transpose the image block when the image block has horizontal directionality, and when the image block has vertical directionality, the processor 120 may transpose the image block. When only vertical directionality is used and horizontal directionality is not used when creating the eigenvector, index matrix, and multiple filters, the processor 120 does not transpose the image block when the image block has vertical directionality, and when the image block has horizontal directionality, the processor 120 may transpose the image block.

[0152] In Figure 3B the base block indicates the directionality used when creating the eigenvector, index matrix, and multiple filters.

[0153] In the case where the base block has vertical directionality, such as Figure 3B the upper end of, the processor 120 may only transpose the right image block of the left two image blocks. In the case where the base block has horizontal directionality, such as Figure 3B the lower end of, the processor 120 may only transpose the right image block of the left two image blocks.

[0154] Since only one of vertical directionality and horizontal directionality is used, the data volume of the eigenvector, index matrix, and multiple filters can be reduced. Additionally, additional calculations depending on the directionality determination and transposition of the image block can be generated, but due to the reduction in the data volume of the eigenvector, index matrix, and multiple filters, the calculation volume in subsequent operations can be reduced. For example, according to the directionality determination operation, the overall calculation volume can be reduced.

[0155] Meanwhile, masks such as Figure 3A Sx and Sy shown in are only examples, and the processor 120 may determine the directionality of the image block by other methods. Additionally, although Figure 3B only shows the 3×3 pixels in a 5×5 image block, this is for illustrating the directionality. The processor 120 may determine the directionality of the image block by applying a 5×5 image block and the mask shown in Figure 3A .

[0156] However, the present disclosure is not limited thereto, and the processor 120 may also use only some pixels in the image block to determine the directionality of the image block to improve the calculation speed. For example, the processor 120 may determine the directionality of the image block by applying the mask for determining 3×3 directionality to the 3×3 pixels except for the pixels near the edge in the 5×5 image block.

[0157] In the following, for the sake of convenience of explanation, the case of inputting a 3×3 image block is described. However, the present disclosure is not limited thereto, and the processor 120 can operate on image blocks of other sizes in the same manner. In addition, the memory 110 can store masks corresponding to various image sizes, index matrices, multiple vectors, gradient calculation operators to be described below, etc.

[0158] As Figure 3C shown, the processor 120 can calculate the horizontal gradient and vertical gradient of a 3×3 image block using the Sobel operator. As Figure 3D shown, the processor 120 can obtain the 3×3 horizontal gradient and 3×3 vertical gradient by multiplying the Sobel operator by the 3×3 image block.

[0159] As Figure 3E shown, the processor 120 can calculate a c×1 gradient vector from the 3×3 horizontal gradient and 3×3 vertical gradient.

[0160] Although the case of calculating the gradient vector after transposing the image block is described above, the present disclosure is not limited thereto. For example, the processor 120 can only determine the directionality of the image block and calculate the horizontal gradient and vertical gradient of the image block. In addition, when the image block needs to be transposed, the processor 120 can change the calculation order of the gradient vector by reflecting the transpose from the horizontal gradient and vertical gradient.

[0161] Figure 4A and Figure 4B are diagrams showing the feature dimension reduction of the gradient vector according to an exemplary embodiment of the present disclosure.

[0162] As Figure 4A shown, the processor 120 can use a c’×c eigenvector to reduce the feature dimension of the c×1 gradient vector. For example, the processor 120 can obtain a c’×1 gradient vector, the feature dimension of which is reduced by the multiplication between the c’×c eigenvector and the c×1 gradient vector.

[0163] Figure 4B shows an example of the gradient vector gradient vector’ whose feature dimension is reduced.

[0164] The eigenvector can be created by an external server through a principal component analysis method, and the electronic device 100 can receive the eigenvector from the external server and store the eigenvector in the memory 110. For example, the external server can create the gradient vector of each of the 10,000 first sample image patches. The method of creating the gradient vector is the same as the method described above. Here, the external server can determine the directionality of each of the first sample image patches, transpose the first sample image patches based on the determined directionality, and calculate the gradient vector for the transposed first sample image patches. Then, the external server can calculate the eigenvector by applying the principal component analysis method to each of the calculated multiple sample gradient vectors.

[0165] However, the present disclosure is not limited thereto, and the processor 120 can also directly create the eigenvector through the principal component analysis method. In this case, the memory 110 can store multiple first sample image patches.

[0166] Hereinafter, for ease of explanation, the case of using the gradient vector with reduced feature dimension is described. However, the reduction of the feature dimension as described above is an optional operation, and thus the subsequent operations can be performed without reducing the feature dimension of the gradient vector.

[0167] Figure 5A and Figure 5B are diagrams illustrating an exemplary search of a filter using an index matrix according to an exemplary embodiment of the present disclosure.

[0168] As Figure 5A shown, the memory 110 can store an index matrix of a×c'. The index matrix can include an index vector of 1×c'. Here, the index vector represents the characteristics of the image patch and can also be referred to as a class or a group. Figure 5A shows an index matrix including an index vector indicating the characteristics of each of the a classes or groups into which the characteristics of the image patch are divided.

[0169] The processor 120 can perform the multiplication between the index matrix of a×c' and the gradient vector of c'×1 with reduced feature dimension, and calculate the correlation vector of a×1 as Figure 5B shown. Each row of the correlation vector indicates the correlation between the image patch and each class or each group.

[0170] The higher the number in the correlation vector, the higher the correlation. For example, in the case where 0.9 is the maximum number in Figure 5B , the processor 120 can determine that the image patch has the characteristics of the class or group corresponding to 0.9.

[0171] The index matrix may be created by an external server, and the electronic device 100 may receive the index matrix from the external server and store the index matrix in the memory 110. The external server may use a K-SVD or K-Means algorithm (one of the unsupervised learning methods) to calculate an index vector, which may represent a plurality of sample gradient vectors calculated from a plurality of first sample image blocks. The external server may create an index matrix including the index vector.

[0172] However, the present disclosure is not limited thereto, and the processor 120 may further calculate an index vector that may represent a plurality of sample gradient vectors calculated from a plurality of first sample image blocks.

[0173] In addition, although the above description describes a case where the characteristics of an image block are divided into a number of classes or groups, the characteristics of an image block are not limited thereto and may be divided into another number of classes or groups. However, for ease of explanation, the following description describes a case where the characteristics of an image block are divided into a number of classes or groups. In addition, while index vectors, classes, and groups are used together in the above description, the following description describes a case where a class is used as a concept for indicating the characteristics of an image block and an index vector indicates each class.

[0174] Meanwhile, the number of classes or groups may vary depending on the type of filter. For example, in the case of a resolution upscaling filter, the characteristics of the image blocks may be divided into a number of classes or groups, but in the case of a sharpening filter, the characteristics of the image blocks may be divided into a number of classes or groups different from a.

[0175] Figure 6A 、 Figure 6B and Figure 6C is a diagram illustrating an exemplary method of reducing the amount of search calculations for a filter according to an exemplary embodiment of the present disclosure.

[0176] Figure 6A 6 is a diagram showing that a plurality of classes are grouped into four groups, and a class 620 representing each group may be specified in each of the groups 610 , 611 , 612 , and 613 . Figure 6B A first index matrix IndexMatrix1 is shown, which includes a plurality of index vectors corresponding to a class 620 representing each group. Figure 6C A second index matrix IndexMatrix2 is shown, which includes a plurality of index vectors corresponding to the classes included in one of the four groups. That is, the number of second index matrices is the same as the number of groups.

[0177] The first index matrix and the second index matrix may be created by an external server, and the electronic device 100 may receive the first index matrix and the second index matrix from the external server and store the first index matrix and the second index matrix in the memory 110. The external server may perform grouping on the index vectors by a k-means clustering method or the like and determine the index vectors representing each group. However, the present disclosure is not limited thereto, and the external server may also calculate the representative index vectors representing each group. In this case, the representative index vectors may not coincide with all the index vectors in the corresponding group.

[0178] However, the present disclosure is not limited thereto, and the processor 120 may also calculate the first index matrix and the second index matrix by the method described above and store the first index matrix and the second index matrix in the memory 110.

[0179] The processor 120 may perform the multiplication between the first index matrix of 4×c’ and the gradient vector of c’×1 with the reduced feature dimension and calculate the correlation with each group. The processor 120 may obtain a group based on the maximum correlation among the correlations with each group. The processor 120 may perform the multiplication between the second index matrix of b×c’ corresponding to the obtained group and the gradient vector of c’×1 with the reduced feature dimension and calculate the correlation with each class in the classes included in the obtained group.

[0180] In this case, the calculation of the correlation of the classes other than the three representative classes among the b×3 classes included in the other groups except the obtained group is not performed, and compared with Figure 5A and Figure 5B the accuracy may be reduced, but the calculation speed may be increased.

[0181] In the case of reducing the number of groups, the accuracy may also be increased. The calculation speed may also be increased by increasing the number of groups.

[0182] Figure 7A and Figure 7B are diagrams illustrating an exemplary method of reducing the search calculation amount of filters according to another exemplary embodiment of the present disclosure. For example, Figure 7A and Figure 7B The method is a method of selecting classes with higher correlation by additional calculation after completely performing the methods of Figure 6A , Figure 6B and Figure 6C .

[0183] As Figure 7A shown, the processor 120 may obtain class A 710, which is the class with the highest correlation with the image block, in the third group as a result of the calculation of the correlation with the second index matrix.

[0184] AsFigure 7B As shown, the memory 110 may store a third index matrix, index matrix 3, for class A 710. The third index matrix may include a plurality of index vectors corresponding to classes A'720-1, A'720-2, and A'720-3 that are included in a group different from class A 710 and have characteristics similar to those of class A 710. Additionally, the memory 110 may store a third index matrix for each class that is not class A 710. For example, the number of third index matrices may be the same as the number of classes. Additionally, the sizes of the plurality of third index matrices may be different from each other.

[0185] The plurality of third index matrices may be created by an external server or by the processor 120.

[0186] The processor 120 may perform the multiplication between a 3×c' third index matrix and a c'×1 gradient vector with reduced feature dimension, and calculate the correlations with classes A'720-1, A'720-2, and A'720-3, which have characteristics similar to those of class A710.

[0187] With Figure 6A 、 Figure 6B and Figure 6C Compared with the methods of Figure 7A and Figure 7B The process of calculating the correlation with the third index matrix may be added to the methods of

[0188] Figure 8A and Figure 8B FIGS. 8 and

[0189] The processor 120 may apply at least one filter from a plurality of filters to an image block based on the correlations calculated by the methods shown in Figure 5A 、 Figure 5B 、 Figure 6A 、 Figure 6B 、 Figure 6C 、 Figure 7A and Figure 7B For example, when as Figure 5BWhen the correlations with all classes are calculated as shown, the processor 120 can obtain the first filter corresponding to the class with a correlation of 0.9, the second filter corresponding to the class with a correlation of 0.2, and the third filter corresponding to the class with a correlation of 0.1 in the order of high correlations. Each filter among the multiple filters can correspond to each class among the multiple classes. For example, in the case of creating a classes based on the characteristics of the image block, one filter corresponding to each of the a classes is created, and a total of a filters can be created. A detailed description of the method for creating filters is described below.

[0190] The processor 120 can convert the multiple filters into one filter through the following Equation 2:

[0191] [Equation 2]

[0192]

[0193] Here, H(1), H(2), and H(3) respectively indicate the first filter, the second filter, and the third filter, and C(1), C(2), and C(3) respectively indicate the correlations between the image block and the first filter, the second filter, and the third filter.

[0194] However, the present disclosure is not limited thereto, and the processor 120 can also convert the multiple filters into one filter without considering the correlations of each filter. In this case, the processor 120 can convert the multiple filters into one filter by replacing C(1), C(2), and C(3) in Equation 2 with 1.

[0195] As Figure 8A shown, the processor 120 can obtain the final image block 820 for the target pixel 810 by applying the filter to the 5×5 image block. Figure 8A The right side of shows the final image block of a total of six target pixels after image processing, and it is a view after applying the resolution magnification filter. That is, one target pixel can be magnified into a total of nine pixels.

[0196] The processor 120 can vectorize the image block and apply the filter to the image block. For example, the processor 120 can vectorize the 5×5 image block and apply the filter to the 5×5 image block. That is, the processor 120 can obtain the 3×3 final image block as shown on the right side of by performing the multiplication between the filter and the vector. The method of vectorizing the image block and obtaining the final image block from the vector is the same or similar to the method described in Figure 8A and thus the overlapping description is omitted. Figure 3E

[0197] ​Although the use of a resolution magnification filter is described above, the present disclosure is not limited thereto. For example, a sharpening filter may be used. In this case, the resolution remains unchanged, and the target pixel may be filtered based on multiple surrounding pixels to be sharpened. The sharpening filter may also be created by an artificial intelligence algorithm, and the sample images used in the learning process may be different from the sample images used in the resolution magnification filter. A detailed description thereof is provided below.

[0198] Meanwhile, the processor 120 may store information on whether the image block is transposed according to the directionality of the initial image block in the memory 110. When obtaining the final image block from the transposed image block, as Figure 8B shown, the processor 120 may transpose the final image block again based on the information stored in the memory 110.

[0199] Meanwhile, the processor 120 may obtain an additional image block by applying a non-learning-based filter to the image block, and update the final image block based on the maximum correlation among the calculated multiple correlations and the additional image block.

[0200] Specifically, the processor 120 may update the final image block as represented by the following Equation 3:

[0201] [Equation 3]

[0202] Y1′ = w·Y1+(1 - w)·Y2

[0203]

[0204] Here, Y1’ indicates the updated final image block, Y1 indicates the final image block, Y2 indicates the additional image block, w indicates the weight, c indicates the maximum correlation among the calculated multiple correlations, and k indicates a constant.

[0205] Depending on the type of the filter and the type of the image characteristics, the non-learning-based filter may exhibit better performance than the learning-based filter. For example, the learning-based resolution magnification filter exhibits excellent performance for the edge components with clear image features, but exhibits noise components for the flat regions with unclear image features or low performance on the surfaces with detailed characterization. Therefore, as represented by the following Equation 3, in the case of updating the final image block obtained by the learning-based resolution magnification filter with the additional image block obtained by the non-learning-based resolution magnification filter, excellent filtering performance may be provided for the flat regions with unclear image features.

[0206] In addition, since the correlation between the image block and the learning-based resolution magnification filter is regarded as a weight during the update process, the performance can be further improved. For example, in the case where the image block is a flat area with unclear image features, the correlation between the image block and multiple classes may be very low. During the learning process, sample image blocks including flat areas with unclear image features can be excluded from the learning.

[0207] Meanwhile, the processor 120 can also perform resolution magnification of a non-learning magnification by connecting a non-learning-based resolution magnification filter in series to the learning-based resolution magnification filter.

[0208] For example, the learning-based resolution magnification filter operates only according to the learned resolution magnification and may not operate on the non-learned resolution magnification. However, the processor 120 can use the learning-based resolution magnification filter to magnify the resolution and then use the non-learning-based resolution magnification filter to perform resampling to obtain a result according to the desired resolution magnification.

[0209] Meanwhile, although the case of weighting and summing three filters in Equation 2 is described, the present disclosure is not limited thereto. For example, the processor 120 can use only one filter or use multiple filters and apply different weights to the multiple filters.

[0210] In addition, the processor 120 can provide a final image block using one filter and multiple final image blocks using multiple filters to which different weights are applied, and can change Equation 2 according to the user's selection. For example, the processor 120 can provide a final image block using one filter and multiple final image blocks using multiple filters according to Equation 2, and in the case where the user selects the final image block using one filter, the processor 120 can use one filter in the subsequent filter application process.

[0211] Figure 9A and Figure 9B are diagrams showing an exemplary learning method of a filter according to an exemplary embodiment of the present disclosure. In Figure 9A and Figure 9B for the sake of convenience of explanation, a learning method of the resolution magnification filter and the case of performing learning in an external server are described.

[0212] As Figure 9A shown, the external server can store high-resolution images. As Figure 9BAs shown, the external server can read multiple second sample image patches from a high-resolution image and obtain multiple first sample image patches by reducing the resolution of each of the multiple second sample image patches. The external server can store the multiple second sample image patches with high resolution and the multiple first sample image patches corresponding to the multiple second sample image patches and having low resolution. Here, high resolution and low resolution are intended to indicate a relative state, and a low-resolution image patch can represent, for example, an image patch having a resolution lower than that of a high-resolution image patch.

[0213] The external server can obtain multiple sample gradient vectors from the multiple first sample image patches. Here, the external server can perform transposition according to the directionality of the multiple first sample image patches, but this is an optional operation.

[0214] The external server can obtain eigenvectors from the multiple sample gradient vectors and reduce the feature dimension of the multiple sample gradient vectors. However, the external server can also perform learning without reducing the feature dimension.

[0215] The external server can divide the multiple sample gradient vectors with reduced feature dimension into a predetermined number of groups for each characteristic of the image patch and obtain index vectors representing each group. Here, the index vectors representing each group can be the same as one corresponding to the respective group among the multiple sample gradient vectors, or can be different. The external server can obtain an index matrix including the multiple index vectors.

[0216] For ease of explanation, a case where the external server divides the characteristics of the image patch into two groups with vertical directionality and horizontal directionality is described. In this case, the external server can obtain two index vectors representing each group and obtain an index matrix including the two index vectors.

[0217] In addition, the external server can classify the first sample image patches into one of the two groups. For example, the external server can calculate the first sample gradient vector of the first sample image patch, calculate the correlation between the first sample image patch and the two groups by multiplying the index matrix and the first sample gradient vector, and classify the first sample image patch into the group with higher correlation. The external server can perform the above-described process for all the first sample image patches. Thus, for example, 354,000 of the 1,000,000 first sample image patches can be classified into the group with horizontal directionality, and 646,000 of the 1,000,000 first sample image patches can be classified into the group with vertical directionality.

[0218] The external server can create the first filter in the following way: by using an artificial intelligence algorithm to learn the relationship between 354,000 first sample image patches with horizontal directivity and 354,000 second sample image patches corresponding to the 354,000 first sample image patches. Additionally, the external server can create the second filter in the following way: by using an artificial intelligence algorithm to learn the relationship between 646,000 first sample image patches with vertical directivity and 646,000 second sample image patches corresponding to the 646,000 first sample image patches.

[0219] The index vector can be considered as an index indicating the characteristics of an image patch, and based on the characteristics of the image patch corresponding to the corresponding index vector, the filter corresponding to the index vector can be obtained by learning the first sample image patch and the second sample image patch corresponding to the first sample image patch.

[0220] The external server can perform learning by methods of minimizing and / or reducing the L2 norm error and methods of minimizing and / or reducing the L1 norm error. For example, the external server can calculate the image magnification filter by minimizing and / or reducing the L2 norm error through the following Equation 4:

[0221] [Equation 4]

[0222]

[0223]

[0224] Here, K refers to the class, X indicates the image value of the high-resolution sample image patch, and Y indicates the image value of the low-resolution sample image patch.

[0225] However, this is only an example, and various learning methods can be used.

[0226] Additionally, although Figure 9A and Figure 9B describe the situation of learning high-resolution image patches and low-resolution image patches, the present disclosure is not limited thereto. For example, the external server can also obtain a sharpening filter or a blurring filter by learning blurred image patches and sharpened image patches. In this case, the resolution of the image patches may remain unchanged. Here, a blurred image patch means an image patch having a relatively lower sharpness than the sharpened image patch.

[0227] In addition, instead of the external server, the electronic device 100 may directly perform learning. In this case, the memory 110 may store the first sample image block and the second sample image block, and the processor 120 may obtain the index matrix and multiple filters by performing learning based on the stored information. In addition, when an input image block is received, the processor 120 may create a final image block by applying at least one filter to the input image block.

[0228] Meanwhile, the transposition of the image block depending on the direction determination result, the reduction of the feature dimension of the gradient vector, and the application of the non-learning-based filter described above are all optional operations. For example, the index matrix and the filter may vary depending on whether the optional operations are performed.

[0229] For example, when the transposition of the image block depending on the direction determination result is performed, an index matrix of a×c may be created (when the feature dimension reduction is not performed). On the other hand, when the transposition of the image block depending on the direction determination result is not performed, an index matrix of 2×a×c may be created (when the feature dimension reduction is not performed). The reason is that when the image characteristics are divided into a, in the case where the transposition is not performed, two cases of horizontal directionality and vertical directionality are divided, so the image characteristics are doubled.

[0230] When the reduction of the feature dimension of the gradient vector is performed, an index matrix of a×c' may be created (when the transposition is performed). On the other hand, when the reduction of the feature dimension of the gradient vector is not performed, an index matrix of a×c may be created (when the transposition is performed).

[0231] Figure 10 FIG. is a flowchart illustrating an exemplary image transmission operation according to an exemplary embodiment of the present disclosure.

[0232] The external server 200 may be, for example but not limited to, a device that provides images to the electronic device 100. In addition, the external server 200 may be, for example but not limited to, a desktop computer, a laptop personal computer, a tablet computer, a smartphone, etc., as well as a server, and may be any device that can perform learning and encoding through artificial intelligence algorithms.

[0233] The external server 200 may obtain a low-resolution image from a high-resolution image (S1010). For example, the external server 200 may obtain a 4K video from an 8K video. The method of reducing the resolution of the video may be any method.

[0234] In addition, the external server 200 can create filter information by learning high-resolution images and low-resolution images (S1020). For example, the external server 200 can divide the corresponding frames of the high-resolution images into image patches with a predetermined size, and can also divide the low-resolution images by the same method. In this case, the number of image patches created from the high-resolution images can be the same as the number of image patches created from the low-resolution images. In addition, each image in the image patches created from the high-resolution images can correspond to one image in the image patches created from the low-resolution images, and can only differ from this one image in the image patches created from the low-resolution images in terms of resolution. Hereinafter, the image patches created from the high-resolution images and the image patches created from the low-resolution images are respectively described as high-resolution image patches and low-resolution image patches.

[0235] The external server 200 can create multiple filters by performing learning for deriving corresponding high-resolution image patches from low-resolution image patches. Here, the method of creating multiple filters is the same as the method Figures 1A to 9B described in, and thus the description of this part is not repeated here.

[0236] In addition, the external server 200 can encode and compress the low-resolution images (S1030). The encoding method is not particularly limited.

[0237] The external server 200 can send the filter information and the encoded low-resolution images to the electronic device 100 (S1040). Here, the filter information can include an index matrix and multiple filters. In addition, according to an exemplary embodiment, the filter information can also include eigenvectors. In addition, the filter information can include a method of dividing the corresponding frames of the low-resolution images into image patches with a predetermined size.

[0238] The electronic device 100 can decode the encoded low-resolution images to reconstruct the low-resolution images (S1050). The decoding method is not particularly limited as long as it corresponds to the encoding method of the external server 200.

[0239] The electronic device 100 can reconstruct the high-resolution images by applying the filter information to the low-resolution images (S1060). For example, the electronic device 100 can divide the low-resolution images into multiple image patches based on the method of dividing the corresponding frames of the low-resolution images into image patches with a predetermined size included in the filter information. In addition, the electronic device 100 can obtain high-resolution image patches from the corresponding image patches based on the index matrix and multiple filters. The method of obtaining high-resolution image patches is the same as the method Figures 1A to 9B described in, and thus the description of this part is not repeated here. The electronic device 100 can obtain the high-resolution images from the multiple high-resolution image patches.

[0240] It is difficult to transmit an 8K image only through current encoding techniques, but an 8K image can be transmitted by the method described above. For example, the external server 200 can encode a 4K image and transmit the encoded 4K image to the electronic device 100. In addition, the electronic device 100 can decode the received information to reconstruct the 4K image, and can create an 8K image from the 4K image using learning data learned through an artificial intelligence algorithm, thereby deriving an 8K image from the 4K image.

[0241] The filter information transmitted from the external server 200 to the electronic device 100 is only a tool for deriving an 8K image from a 4K image, and can be considered as information different from the image information. For example, the capacity of the filter information can be much smaller than the capacity when directly transmitting an 8K image, and in the case of using the filter information described above, streaming of the 8K image is feasible.

[0242] Figure 11 It is a flowchart showing an exemplary method of controlling an electronic device according to an exemplary embodiment of the present disclosure.

[0243] Based on the relationship between pixels within an image block, classify an image block including a target pixel and a plurality of surrounding pixels into one of a plurality of image patterns (S1110). In addition, obtain a final image block by applying at least one filter corresponding to the classified image pattern among a plurality of filters each corresponding to a plurality of image patterns to the image block, and in the final image block, the target pixel has undergone image processing (S1120). The plurality of filters can be obtained by: based on each of the plurality of image patterns, learning the relationship between a plurality of first sample image blocks and a plurality of second sample image blocks corresponding to the plurality of first sample image blocks through an artificial intelligence algorithm.

[0244] The classification (S1110) can include calculating a gradient vector from the image block, calculating the correlation between the image block and each of a plurality of index vectors based on the gradient vector and an index matrix including a plurality of index vectors, and classifying the image block into one of the plurality of image patterns based on the calculated correlation, and the plurality of index vectors can be obtained based on a plurality of sample gradient vectors calculated from a plurality of first sample image blocks, and the plurality of index vectors can respectively correspond to the plurality of image patterns.

[0245] In addition, during the process of calculating the gradient vector: when the intensity of the horizontal directionality of the image block is greater than the intensity of the vertical directionality of the image block, a gradient vector can be calculated from the image block; and when the intensity of the horizontal directionality of the image block is less than the intensity of the vertical directionality of the image block, the image block can be transposed and a gradient vector can be calculated from the transposed image block.

[0246] Meanwhile, the method of controlling an electronic device according to an exemplary embodiment of the present disclosure may further include reducing the size of the gradient vector using an eigenvector, wherein, in the process of calculating the correlation, the correlation between the image block and each of the plurality of index vectors is calculated based on the gradient vector with the reduced size, and the eigenvector is created by applying principal component analysis of a plurality of sample gradient vectors.

[0247] In addition, calculating the correlation may include calculating a first correlation between the image block and each of a plurality of index vector groups based on a first index matrix and the gradient vector, obtaining one index vector group from the plurality of index vector groups based on the first correlation, obtaining a second index matrix corresponding to the obtained index vector group from among a plurality of second index matrices, and calculating a second correlation between the image block and each of the plurality of index vectors included in the obtained index vector group based on the gradient vector and the obtained second index matrix, and in the process of obtaining the final image block (S1120), the final image block may be obtained by applying at least one filter from a plurality of filters to the image block based on the second correlation, the plurality of index vectors may be divided into a predetermined number of groups to obtain the plurality of index vector groups, the first index matrix may include index vectors representing each of the plurality of index vector groups, and each of the plurality of second index matrices may include a plurality of index vectors corresponding to each of the plurality of index vector groups.

[0248] Calculating the correlation may include obtaining one index vector from the plurality of index vectors included in the obtained index vector based on the second correlation, obtaining at least one additional index vector included in other index vector groups among the plurality of index vector groups and corresponding to the obtained index vector based on similar index vector information, and calculating a third correlation between the image block, the obtained index vector, and the obtained additional index vector, and in the process of obtaining the final image block (S1120), the final image block may be obtained by applying at least one filter from a plurality of filters to the image block based on the third correlation.

[0249] Meanwhile, the process of obtaining the final image block (S1120) may include obtaining at least two filters from the plurality of filters based on the calculated plurality of correlations, calculating a final filter based on the at least two filters and the correlation corresponding to each of the at least two filters, and obtaining the final image block by applying the calculated final filter to the image block.

[0250] In addition, the plurality of first sample image blocks may be images with the resolutions of the corresponding second sample image blocks reduced respectively, and in the process of obtaining the final image block (S1120), the final image block with the resolution of the target pixel enlarged may be obtained by applying at least one filter among the plurality of filters to the image block.

[0251] The method for controlling an electronic device according to an exemplary embodiment of the present disclosure may further include obtaining an additional image block with the resolution of the target pixel enlarged by applying a non-learning-based resolution magnification technique to the image block, and updating the final image block based on the maximum correlation among the calculated plurality of correlations and the additional image block.

[0252] Meanwhile, in the process of calculating the gradient vector, the horizontal gradient and the vertical gradient of each of the plurality of surrounding pixels and the target pixel may be calculated, and the gradient vector may be calculated based on the horizontal gradient and the vertical gradient.

[0253] In addition, each of the plurality of filters may be obtained by: obtaining a plurality of first sub-sample image blocks corresponding to one index vector among the plurality of index vectors from among the plurality of first sample image blocks, obtaining a plurality of second sub-sample image blocks corresponding to the plurality of first sub-sample image blocks from among the plurality of second sample image blocks, and learning the relationship between the plurality of first sub-sample image blocks and the plurality of second sub-sample image blocks through an artificial intelligence algorithm.

[0254] According to the various exemplary embodiments of the present disclosure described above, the electronic device may perform image processing in real time and thus be used in various applications, and the electronic device may improve the quality of image processing compared with non-learning-based image processing by performing learning-based image processing.

[0255] Meanwhile, the processor may apply a filter to each pixel of the image block. The processor may apply a filter to each sub-pixel of each pixel in the image block. The processor may apply a filter to only a part of the sub-pixels. The processor may distinguish the chrominance component and the luminance component of each pixel in the image block from each other, and apply the filter to at least one of the chrominance component and the luminance component.

[0256] Meanwhile, according to an exemplary embodiment of the present disclosure, the various exemplary embodiments described above can be implemented by software, which includes instructions stored in a machine-readable storage medium (e.g., a computer-readable storage medium). A machine can be a device that calls the stored instructions from the storage medium and can operate according to the called instructions, and can include an electronic device according to an exemplary embodiment of the present disclosure. In the case where the command is executed by a processor, the processor can directly execute the function corresponding to the command, or other components can execute the function corresponding to the command under the control of the processor. The command can include code created or executed by a compiler or an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Here, the term "non-transitory" can refer to, for example, a tangible storage medium that does not distinguish whether data is stored semi-permanently or temporarily on the storage medium.

[0257] In addition, according to an exemplary embodiment of the present disclosure, the methods according to the various exemplary embodiments described above can be included and provided in a computer program product. The computer program product can be traded as a product between a seller and a buyer. The computer program product can be distributed in the form of a storage medium (e.g., a compact disc read-only memory (CD-ROM)), which can be read by a machine or read online through an application store (e.g., PlayStore TM ). In the case of online distribution, at least a part of the computer program product can be stored at least temporarily in a storage medium, such as the memory in the following servers: the manufacturer's server, the application store's server, or a relay server temporarily created.

[0258] In addition, according to an exemplary embodiment of the present disclosure, the various exemplary embodiments described above can be implemented in a computer or a computer-readable recording medium using software, hardware, or any combination of software and hardware. In some cases, the exemplary embodiments described in the present disclosure can be implemented by the processor itself. According to the software embodiment, the exemplary embodiments such as programs and functions described in the present disclosure can be implemented by separate software modules. Each software module can execute one or more functions and operations described in the present disclosure.

[0259] Meanwhile, computer instructions for performing the processing operations of the machine according to the various exemplary embodiments of the present disclosure may be stored in a non-transitory computer-readable medium. The computer instructions stored in the non-transitory computer-readable medium allow a specific machine to perform processing operations in the machine according to the various example embodiments when they are executed by a processor of the specific machine. The non-transitory computer-readable medium may refer to, for example, a medium that semi-permanently stores data and is machine-readable. Examples of the non-transitory computer-readable medium may include optical discs (CDs), digital versatile discs (DVDs), hard disks, Blu-ray discs, universal serial buses (USBs), memory cards, read-only memories (ROMs), and the like.

[0260] In addition, each component (e.g., module or program) according to the various exemplary embodiments described above may include a single entity or multiple entities, and some of the corresponding sub-components described above may be omitted, or the various exemplary embodiments may further include other sub-components. Alternatively or additionally, some components (e.g., modules or programs) may be integrated into one entity and may perform functions in the same or similar manner as those performed by the corresponding components before being integrated. The operations performed by the modules, programs, or other components according to the various exemplary embodiments may be performed in a sequential manner, a parallel manner, an iterative manner, or a heuristic manner, at least some of the operations may be performed in a different order or omitted, or other operations may be added.

[0261] Although the various exemplary embodiments of the present disclosure have been shown and described above, the present disclosure is not limited to the above specific exemplary embodiments, but various modifications may be made by those skilled in the art to the present disclosure without departing from the true spirit and full scope of the present disclosure as disclosed in the appended claims. These modifications should also be understood to fall within the scope and spirit of the present disclosure.

Claims

1. Electronic devices, including: a memory configured to store a plurality of filters, wherein the plurality of filters correspond to a plurality of image patterns respectively; as well as Processor, configured as: determining whether to transpose the image block based on a directionality of an image block including a target pixel and a plurality of pixels surrounding the target pixel; classifying the image block or the transposed image block into one of the plurality of image patterns based on a relationship between the target pixel and the plurality of pixels surrounding the target pixel; as well as obtaining a final image block by applying at least one filter corresponding to the classified image pattern among the plurality of filters to the image block, wherein a resolution of the target pixel is magnified; The multiple filters are obtained by learning the relationship between multiple first sample image blocks and multiple second sample image blocks, wherein the multiple first sample image blocks are obtained by respectively reducing the resolution of the multiple second sample image blocks.

2. The electronic device according to claim 1, wherein The memory is configured to store an index matrix, wherein the index matrix includes a plurality of index vectors respectively corresponding to the plurality of image patterns; The processor is configured to: determining a gradient vector from the image block; determining, based on the gradient vector and the index matrix, a correlation between the image block and each index vector of the plurality of index vectors; as well as classifying the image block into one of the plurality of image patterns based on the determined correlation; The plurality of index vectors respectively represent a plurality of groups, in which a plurality of sample gradient vectors obtained from the plurality of first sample image blocks are divided based on a feature of each of the plurality of first sample image blocks.

3. The electronic device according to claim 2, wherein: The processor is configured to: determining the gradient vector from the image block when the intensity of the horizontal directionality of the image block is greater than the intensity of the vertical directionality of the image block; and When the intensity of the horizontal directivity of the image block is smaller than the intensity of the vertical directivity of the image block, the image block is transposed and the gradient vector is determined from the transposed image block.

4. The electronic device according to claim 2, wherein: The processor is configured to reduce the size of the gradient vector using the eigenvectors stored in the memory, and determine a correlation of the image block with each of the plurality of index vectors based on the reduced-size gradient vector, and The eigenvectors are created by applying principal component analysis of the plurality of sample gradient vectors.

5. The electronic device according to claim 2, wherein: The processor is configured to determine a first correlation between the image block and each index vector group in a plurality of index vector groups based on the gradient vector and a first index matrix stored in the memory, obtain one index vector group from the plurality of index vector groups based on the first correlation, determine a second correlation between the image block and each index vector in a plurality of index vectors included in the obtained index vector group based on the gradient vector and a second index matrix corresponding to the obtained index vector group from among a plurality of second index matrices stored in the memory, and obtain the final image block by applying at least one filter from the plurality of filters to the image block based on the second correlation, wherein: The plurality of index vector groups are obtained by dividing the plurality of index vectors into a predetermined number of groups, The first index matrix includes an index vector representing each index vector group in the plurality of index vector groups, and Each second index matrix in the plurality of second index matrices includes a plurality of index vectors corresponding to each index vector group in the plurality of index vector groups.

6. The electronic device according to claim 5, wherein: The processor is configured to obtain one of the plurality of index vectors included in the obtained index vector group based on the second correlation, obtain at least one additional index vector included in the plurality of index vector groups and corresponding to the obtained index vector other than one of the plurality of index vectors obtained based on similar index vector information stored in the memory, determine a third correlation of the image block based on the gradient vector, the obtained index vector, and the obtained additional index vector, and obtain the final image block by applying at least one filter of the plurality of filters to the image block based on the third correlation.

7. The electronic device according to claim 2, wherein: The processor is configured to obtain at least a first filter and a second filter of the plurality of filters based on the determined correlation, determine a final filter by weighting and summing the first filter and the second filter based on the correlation corresponding to each of the first filter and the second filter, and obtain the final image block by applying the determined final filter to the image block.

8. The electronic device according to claim 1, wherein: The processor is configured to: obtaining an additional image block by applying a non-learning-based resolution upscaling technique to the image block, wherein a resolution of the target pixel is upscaled in the additional image block; as well as The final image block is updated based on the maximum correlation among the determined plurality of correlations and the additional image block.

9. The electronic device according to claim 2, wherein: The electronic device is configured to obtain each of the multiple filters in the following manner: obtaining a plurality of first subsample image blocks corresponding to one index vector among the multiple index vectors from the multiple first sample image blocks, obtaining a plurality of second subsample image blocks corresponding to the multiple first subsample image blocks from the multiple second sample image blocks, and learning the relationship between the multiple first subsample image blocks and the multiple second subsample image blocks through an artificial intelligence algorithm.

10. A method for controlling an electronic device, comprising: determining whether to transpose the image block based on a directionality of an image block including a target pixel and a plurality of pixels surrounding the target pixel; classifying the image block or the transposed image block into one of a plurality of image patterns based on a relationship between the target pixel and the plurality of pixels surrounding the target pixel; as well as obtaining a final image block by applying at least one filter corresponding to the classified image pattern among a plurality of filters each corresponding to the plurality of image patterns to the image block, in which a resolution of the target pixel is enlarged, The multiple filters are obtained by learning the relationship between multiple first sample image blocks and multiple second sample image blocks, wherein the multiple first sample image blocks are obtained by respectively reducing the resolution of the multiple second sample image blocks.

11. The method according to claim 10, wherein: The categories include: determining a gradient vector from the image block; determining a correlation between the image block and each of the plurality of index vectors based on the gradient vector and an index matrix including a plurality of index vectors respectively corresponding to the plurality of image patterns; and classifying the image block into one of the plurality of image patterns based on the determined correlation; The plurality of index vectors respectively represent a plurality of groups, in which a plurality of sample gradient vectors obtained from the plurality of first sample image blocks are divided based on a feature of each of the plurality of first sample image blocks.

12. The method of claim 11, wherein: In the process of determining the gradient vector: calculating the gradient vector from the image block when the intensity of the horizontal directivity of the image block is greater than the intensity of the vertical directivity of the image block; and When the intensity of the horizontal directivity of the image block is smaller than the intensity of the vertical directivity of the image block, the image block is transposed and the gradient vector is calculated from the transposed image block.

13. The method of claim 11 , further comprising reducing the size of the gradient vector using eigenvectors, in, In determining the correlation, a correlation between the image block and each of the plurality of index vectors is calculated based on the gradient vector of reduced size, and The eigenvectors are created by applying a principal component analysis of the plurality of sample gradient vectors.

14. The method of claim 11, wherein: Determining the correlation includes: determining a first correlation between the image block and each index vector group in a plurality of index vector groups based on a first index matrix and the gradient vector; obtaining an index vector group among the plurality of index vector groups based on the first correlation; determining a second correlation between the image block and each of the plurality of index vectors included in the obtained index vector group based on the gradient vector and a second index matrix corresponding to the obtained index vector group among the plurality of second index matrices; and Wherein, in the process of obtaining the final image block, the final image block is obtained by applying at least one filter of the plurality of filters to the image block based on the second correlation, wherein: The plurality of index vector groups are obtained by dividing the plurality of index vectors into a predetermined number of groups, The first index matrix includes an index vector representing each index vector group in the plurality of index vector groups, and Each second index matrix in the plurality of second index matrices includes a plurality of index vectors corresponding to each index vector group in the plurality of index vector groups.

Images