Image processing apparatus and operating method thereof

Through neural network technology, image processing is performed using convolutional neural network and multi-layer perceptron modules, the problem of noise removal and quality improvement in image data processing is solved, and automated and efficient image improvement is achieved.

CN119072712APending Publication Date: 2024-12-03SAMSUNG ELECTRONICS CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202380033847.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2022-10-05
Filing Date
2023-04-06
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

The prior art is difficult to process image data efficiently, especially in terms of removing noise and improving image quality and resolution, human intelligence is insufficient to cope with the growth of data traffic.

Method used

Using neural networks, including convolutional neural networks and multi-layer perceptron modules, high-quality and high-resolution images are generated through technologies such as feature extraction, upsampling, adaptive instance normalization and spatial feature transformation.

Benefits of technology

Automatic image processing is implemented, enabling the removal of noise while maintaining fine edges and textures of the image, generating images of higher quality and resolution than the original image.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119072712A_ABST
    Figure CN119072712A_ABST
Patent Text Reader

Abstract

An image processing apparatus for processing an image by using one or more neural networks according to one embodiment may include: a memory for storing one or more instructions; and at least one processor that obtains classification information and first feature information about the first image by executing the one or more instructions; performing first image processing on the classification information and the first feature information to generate a first feature image of the first image; obtaining second feature information by performing second image processing on the classification information and the first feature information; obtaining fourth feature information by performing third image processing on third feature information extracted during the first image processing; generating a second feature image of the first image based on the second feature information and the fourth feature information; and generating a second image based on the first feature image and the second feature image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus for processing an image by using a neural network, and a method for operating the image processing apparatus. Background Art

[0002] As data traffic (e.g., image data) grows exponentially with the development of computer technology, the human intelligence that processes data (e.g., manually or through rule-based methods) is insufficient. Artificial intelligence has become an important trend that will drive future innovation. Because artificial intelligence imitates human thinking, it is infinitely applicable to almost all industries. Representative technologies of artificial intelligence include pattern recognition, machine learning, expert systems, neural networks, natural language processing, etc.

[0003] For example, a neural network simulates the characteristics of human biological nerve cells by using mathematical expressions, and uses an algorithm that imitates human learning ability. Through the algorithm, the neural network is able to generate a mapping between input data and output data, and the ability to generate such a mapping can be called the learning ability of the neural network. In addition, the neural network has the ability to generate correct output data for input data that has not been used for training based on the training results.

[0004] It may be advantageous to apply artificial intelligence techniques (e.g., neural networks) to process image data. Such artificial intelligence techniques may enable image processing tasks to be performed automatically without human intervention. Image processing tasks may include removing noise or artifacts from an image, improving the quality of an image, or increasing the resolution of an image. Summary of the invention

[0005] Technical Solution Systems and methods for processing images using one or more neural networks are provided. For example, the systems and methods enable the removal of noise or artifacts from an image, the improvement of the quality of an image, or the improvement of the resolution of an image.

[0006] According to an aspect of the present disclosure, an image processing apparatus may include a memory storing one or more instructions and at least one processor configured to execute the one or more instructions stored in the memory.

[0007] The at least one processor may be further configured to execute the one or more instructions stored in the memory to obtain classification information and first feature information of the first image.

[0008] The at least one processor may be further configured to execute the one or more instructions stored in the memory to generate a first feature image of the first image by performing first image processing on the classification information and the first feature information.

[0009] The at least one processor may be further configured to execute the one or more instructions stored in the memory to obtain second feature information by performing second image processing on the classification information and the first feature information.

[0010] The at least one processor may be further configured to execute the one or more instructions stored in the memory to obtain fourth feature information by performing third image processing on third feature information extracted during the first image processing.

[0011] The at least one processor may be further configured to execute the one or more instructions stored in the memory to generate a second feature image of the first image based on the second feature information and the fourth feature information.

[0012] The at least one processor may be further configured to execute the one or more instructions stored in the memory to generate a second image based on the first feature image and the second feature image.

[0013] The first feature image may include features of non-high frequency components included in the first image, and the second feature image may include features of high frequency components included in the first image.

[0014] The second image may be of higher quality than the first image.

[0015] The at least one processor may be further configured to execute the one or more instructions stored in the memory to obtain classification information and first feature information of the first image by using one or more convolutional neural networks.

[0016] The first image processing may include upsampling the first feature information, and the first image, the first feature image, and the second feature image all have the same size.

[0017] The second image processing may be performed by a multilayer perceptron (MLP) module comprising one or more fully connected layers, and the at least one processor may also be configured to execute the one or more instructions stored in the memory to: perform a multiplication operation between input data fed to the fully connected layer and a weight matrix included in the fully connected layer, and an addition operation with a bias included in the fully connected layer.

[0018] The at least one processor may also be configured to execute the one or more instructions stored in the memory to: obtain sub-feature information by performing downscaling and upscaling on the third feature information; obtain difference information between the sub-feature information and the third feature information; and generate fourth feature information by performing a convolution operation between the difference information and a pre-trained weight.

[0019] The at least one processor may also be configured to execute the one or more instructions stored in the memory to: obtain fifth feature information by performing a first operation on the second feature information; and generate a second feature image by performing a second operation on the fifth feature information and the fourth feature information.

[0020] The first operation may include an adaptive instance normalization (AdaIn) operation, and the second operation may include a spatial feature transform (SFT) operation.

[0021] According to an aspect of the present disclosure, an operating method of an image processing device for processing an image by using at least one neural network may include obtaining classification information and first feature information of a first image.

[0022] An operating method of an image processing apparatus for processing an image by using at least one neural network may include generating a first feature image of a first image by performing first image processing on classification information and first feature information.

[0023] The operating method of the image processing apparatus for processing an image by using at least one neural network may include obtaining second feature information by performing second image processing on the classification information and the first feature information.

[0024] The operating method of the image processing apparatus for processing an image by using at least one neural network may include obtaining fourth feature information by performing third image processing on third feature information extracted during the first image processing.

[0025] An operating method of an image processing apparatus for processing an image by using at least one neural network may include generating a second feature image of the first image based on the second feature information and the fourth feature information.

[0026] An operating method of an image processing apparatus for processing an image by using at least one neural network may include generating a second image based on the first feature image and the second feature image.

[0027] The first feature image may include features of non-high frequency components included in the first image, and the second feature image may include features of high frequency components included in the first image.

[0028] The second image may be of higher quality than the first image.

[0029] Obtaining the classification information and the first feature information of the first image may include obtaining the classification information and the first feature information of the first image by using one or more convolutional neural networks.

[0030] The first image processing may include up-sampling the first feature information, and the first image, the first feature image, and the second feature image all have the same size.

[0031] The second image processing may be performed by a multilayer perceptron (MLP) module including one or more fully connected layers, and obtaining the second feature information by performing the second image processing on the classification information and the first feature information may include: performing a multiplication operation between the input data fed to the fully connected layer and the weight matrix included in the fully connected layer, and an addition operation with the bias included in the fully connected layer.

[0032] Obtaining fourth feature information by performing third image processing on third feature information extracted during first image processing may include: obtaining sub-feature information by performing downscaling and upscaling on the third feature information; obtaining difference information between the third feature information and the sub-feature information; and generating fourth feature information by performing a convolution operation between the difference information and a pre-trained weight.

[0033] Generating a second feature image of the first image based on the second feature information and the fourth feature information may include: obtaining fifth feature information by performing a first operation on the second feature information; and generating the second feature image by performing a second operation on the fifth feature information and the fourth feature information.

[0034] The first operation may include an adaptive instance normalization (AdaIn) operation, and the second operation may include a spatial feature transform (SFT) operation.

[0035] The third feature information may include a plurality of pieces of intermediate data that may be output through a process of generating the first feature image.

[0036] According to one aspect of the present disclosure, a non-transitory computer-readable recording medium stores computer-readable program codes or instructions, wherein the computer-readable program codes or instructions can be executed by a processor to perform an image processing method. The method includes: obtaining classification information and first feature information of a first image; generating a first feature image of the first image by performing first image processing on the classification information and the first feature information; obtaining second feature information by performing second image processing on the classification information and the first feature information; obtaining fourth feature information by performing third image processing on third feature information extracted during the first image processing; generating a second feature image of the first image based on the second feature information and the fourth feature information; and generating a second image based on the first feature image and the second feature image. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which: Figure 1 is a diagram illustrating an operation of an image processing apparatus for processing an image by using an image processing network according to an embodiment of the present disclosure; Figure 2 is a diagram illustrating an image processing network according to an embodiment of the present disclosure; Figure 3 is a diagram showing a first module according to an embodiment of the present disclosure; Figure 4 is a diagram illustrating a first feature extractor according to an embodiment of the present disclosure; Figure 5 is a diagram illustrating a first feature image generator according to an embodiment of the present disclosure; Figure 6 is a diagram illustrating a method of processing classification information and first feature information of a first image performed by a third module according to an embodiment of the present disclosure; Figure 7 is a diagram illustrating a method of processing third feature information performed by a third module according to an embodiment of the present disclosure; Figure 8 is a diagram illustrating a second module according to an embodiment of the present disclosure; Fig. 9 is a flowchart illustrating an operation method of an image processing device according to an embodiment of the present disclosure; Fig.10 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0038] Throughout this disclosure, the expression "at least one of a, b, or c" refers to only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0039] The terms used in the specification will now be briefly described, and then the present disclosure will be described in detail.

[0040] As the terms used herein, currently widely used general terms are selected by considering the functions according to the present disclosure, but these terms may have different meanings according to the intention of ordinary technicians in the field, precedents, or the emergence of new technologies. In addition, specific terms may be arbitrarily selected by the applicant, and in this case, the meaning of the selected terms will be described in detail in the detailed description of the present disclosure. Therefore, the terms used herein should not be defined by their simple titles, but are defined based on the meaning of the terms and the overall description of the present disclosure.

[0041] Throughout the specification, when a part "includes" or "contains" an element, unless there is a specific description to the contrary, it should be understood that the part may further include other elements without excluding other elements. In addition, terms such as "part", "module", etc. described in the specification refer to a unit for processing at least one function or operation, and can be implemented as hardware or software, or a combination of hardware and software.

[0042] Embodiments of the present disclosure will be described more fully below with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, embodiments of the present disclosure may have different forms and should not be construed as being limited to the embodiments set forth herein. In addition, parts not related to the description of the present disclosure are omitted to clearly explain embodiments of the present disclosure in the accompanying drawings, and the same reference numerals always represent the same elements.

[0043] Figure 1 A diagram illustrating an operation of an image processing apparatus for processing an image by using an image processing network according to an embodiment of the present disclosure.

[0044] Reference Figure 1 , the image processing network 103 may receive the first image 101 and process the first image 101 to generate the second image 102. In this case, the first image 101 may be an image including noise or artifacts, and may be a low-resolution image or a low-quality image. The image processing apparatus 100 according to an embodiment of the present disclosure may use the image processing network 103 to generate the second image 102 by performing denoising to remove noise while maintaining fine edges and textures of the first image 101. The second image 102 may have a higher quality than the first image 101. The second image 102 may have a higher resolution than the first image 101. However, embodiments of the present disclosure are not limited thereto.

[0045] According to an embodiment of the present disclosure, the image processing network 103 may include one or more neural networks, and for example include a first module, a second module, a third module, and an image synthesizer. However, the embodiment of the present disclosure is not limited thereto.

[0046] In the following, reference will be made to Figures 2 to 8 The image processing network 103 according to an embodiment of the present disclosure is described in detail.

[0047] Figure 2 is a diagram illustrating an image processing network according to an embodiment of the present disclosure.

[0048] Reference Figure 2 , the image processing network 103 may include a first module 210 , a second module 220 , a third module 230 and an image synthesizer 240 .

[0049] According to an embodiment of the present disclosure, the first image x may be input to the first module 210. The first module 210 may obtain first feature information and classification information of the first image x.

[0050] The first feature information and classification information of the first image x obtained by the first module 210 may be input to the third module 230 .

[0051] In addition, the first module 210 may generate a first feature image y1 of the first image x based on the first feature information and the classification information of the first image x. For example, the first module 210 may generate the first feature image y1 of the first image x by performing first image processing on the first feature information and the classification information of the first image x. The first feature image y1 may be an image including features of non-high-frequency components (e.g., low-frequency components and medium-frequency components) included in the first image x. However, embodiments of the present disclosure are not limited thereto.

[0052] In addition, the first module 210 may obtain third feature information extracted during the first image processing, and the third feature information may be input to the third module 230 .

[0053] The third module 230 may obtain the second feature information based on the first feature information and the classification information of the first image x. The third module 230 may generate the second feature information by performing the second image processing on the first feature information and the classification information of the first image x. The second feature information may be input to the second module 220.

[0054] The third module 230 may obtain fourth feature information based on the third feature information. The third module 230 may generate fourth feature information by performing third image processing on the third feature information.

[0055] The second feature information and the fourth feature information may be input to the second module 220 .

[0056] The second module 220 may generate a second feature image y2 of the first image x based on the second feature information and the fourth feature information. The second feature image y2 may be an image including features of high frequency components included in the first image x. However, embodiments of the present disclosure are not limited thereto.

[0057] The image synthesizer 240 may generate the second image y by synthesizing the first feature image y1 and the second feature image y2.

[0058] Hereinafter, a detailed configuration of each of the first module 210 , the second module 220 , and the third module 230 will be described in detail with reference to the accompanying drawings.

[0059] Figure 3 is a diagram illustrating a first module according to an embodiment of the present disclosure.

[0060] Reference Figure 3 , the first module 210 may include a first feature extractor 310 and a first feature image generator 320.

[0061] According to an embodiment of the present disclosure, the first feature extractor 310 may include a feature extraction network, and the feature extraction network may include one or more convolutional layers. The first image x may be input to the first feature extractor 310, and the first feature extractor 310 may obtain classification information F of the first image x by using the feature extraction network. class and the first feature information F latent .

[0062] Will refer to Figure 4 A detailed configuration of the first feature extractor 310 is described.

[0063] Figure 4 is a diagram illustrating a first feature extractor according to an embodiment of the present disclosure.

[0064] Reference Figure 4 , the first feature extractor 310 may obtain the first feature data by performing a convolution operation between the first image x and the first weight included in the first convolution layer 410. The first feature extractor 310 may obtain the second feature data by performing a convolution operation between the first feature data and the second weight included in the second convolution layer 420.

[0065] In addition, the first feature extractor 310 may obtain third feature data by performing a convolution operation between the first image x and the third weight included in the third convolution layer 430 .

[0066] Furthermore, the first feature extractor 310 may perform an element-by-element addition operation between the second feature data and the third feature data at the addition layer 440 .

[0067] According to an embodiment of the present disclosure, the first feature extractor 310 may include a structure in which a module 401 including first, second and third convolutional layers 410, 420 and 430 and an addition layer 440 are repeatedly arranged in series. However, embodiments of the present disclosure are not limited thereto.

[0068] Therefore, the first feature extractor 310 can obtain the classification information F of the first image x. class and the first feature information F latent In this case, the classification information F of the first image x is class The classification information F of the first image x may include data indicating the probability that the first image x is included in each of the preset multiple categories. class and the first feature information F latentEach may be data in a vector form. However, the embodiments of the present disclosure are not limited thereto.

[0069] Return to reference Figure 3 , the classification information F of the first image x class and the first feature information F latent Can be input to the third module 230.

[0070] In addition, the first feature image generator 320 may be based on the classification information F of the first image x obtained by the first feature extractor 310. class and the first feature information F latent The first feature image generator 320 may generate the first feature image y1 by using a feature image generation network. The feature image generation network may include one or more convolution layers.

[0071] Will refer to Figure 5 A detailed configuration of the first feature image generator 320 is described in detail.

[0072] Figure 5 is a diagram illustrating a first feature image generator according to an embodiment of the present disclosure.

[0073] Reference Figure 5 The first feature image generator 320 may perform the classification information F including the first image class and the first feature information F latent The fourth feature data is obtained by performing a convolution operation between the input data and the fourth weight included in the fourth convolutional layer 510.

[0074] The first feature image generator 320 may perform upsampling on the fourth feature data in the first upsampling layer 520. For example, the first feature image generator 320 may perform upsampling by using an interpolation method such as bicubic interpolation or bilinear interpolation. Alternatively, the first feature image generator 320 may perform upsampling using a convolution operation. However, embodiments of the present disclosure are not limited thereto.

[0075] The first feature image generator 320 may obtain fifth feature data by performing a convolution operation between the upsampled data and the fifth weight included in the fifth convolution layer 530 .

[0076] The first feature image generator 320 may generate a first feature image including the classification information F in the second upsampling layer 540. class and the first feature information F latent Upsampling is performed on the input data.

[0077] For example, the first feature image generator 320 may perform upsampling by using an interpolation method such as bicubic interpolation or bilinear interpolation. Alternatively, the first feature image generator 320 may perform upsampling using a convolution operation. However, the embodiments of the present disclosure are not limited thereto.

[0078] The first feature image generator 320 may obtain sixth feature data by performing a convolution operation between the up-sampled data and the sixth weight included in the sixth convolution layer 550 .

[0079] The first feature image generator 320 may perform an element-by-element addition operation between the fifth feature data and the sixth feature data in the addition layer 560 .

[0080] According to an embodiment of the present disclosure, the first feature image generator 320 may include a structure in which a module 501 including a fourth convolution layer 510, a first upsampling layer 520, a fifth convolution layer 530, a second upsampling layer 540, a sixth convolution layer 550, and an addition layer 560 is repeatedly arranged in series. However, embodiments of the present disclosure are not limited thereto.

[0081] The first feature image generator 320 may input, to the third module 230 , third feature information output from each module 501 in a structure in which the modules 501 are repeatedly arranged in series.

[0082] Figure 6 and Figure 7 is a diagram illustrating a third module according to an embodiment of the present disclosure.

[0083] Figure 6 is a diagram showing the processing of the classification information F of the first image performed by the third module according to an embodiment of the present disclosure class and the first feature information F latenent Schematic diagram of the method.

[0084] Reference Figure 6 The third module 230 can apply the Softmax function to the classification information F of the first image input from the first module 210. class To obtain the first input data. For example, the first input data may be calculated by using the following equation 1.

[0085] [Equation 1]

[0086] where p i represents the elements included in the first input data, and z i Indicates the classification information F included in the first image class In addition, T represents the temperature parameter applied to the softmax operation. When the temperature parameter T is large, the classification information F of the first imageclass It can be converted from hard labels to soft labels. For example, when the classification information F of the first image class When converted into a hard label, one of the elements included in the first input data may be represented as 1, and all other elements may be represented as 0. In this case, since an element having a value of 1 indicates that the probability that the first image is included in the category corresponding to the element is 1, the hard label may mean that the first image is classified into one of a plurality of preset categories.

[0087] On the other hand, when the classification information F class When converted into a soft label, a plurality of elements have values ​​other than 0, and each element represents the probability that the first image is included in the category corresponding to the element. Therefore, the soft label may represent the probability that the first image is included in each of the plurality of categories, and when a plurality of objects are included in the first image, the classification information F of the first image is converted into a soft label. class Converting to soft labels can further improve the accuracy of image processing. However, the embodiments of the present disclosure are not limited thereto.

[0088] The third module 230 may obtain the first data by performing a multiplication operation between the first input data obtained from the conversion through the softmax operation and the first weight matrix included in the first linear layer 610 .

[0089] The third module 230 may splice the first feature information F of the first image input from the first module 210 in the splicing layer 620. latent and the first data output from the first linear layer 610. For example, when the first feature information F latent When the first data is a 512×1 vector and the first data is a 512×1 vector, the third module 230 can concatenate the first feature information F latent and the first data to obtain the second data in the form of a 1024×1 vector.

[0090] The third module 230 may obtain third data by performing a multiplication operation between the second data and the second weight matrix included in the second linear layer 630 .

[0091] The third module 230 may normalize the third data output from the second linear layer 630, for example, the third module 230 may normalize the third data so that the sum of the third data is equal to 1. However, the embodiments of the present disclosure are not limited thereto, and various normalization techniques may be used to normalize the third data. The fourth data obtained by normalizing the third data may be input to a multilayer perceptron (MLP) module.

[0092] According to an embodiment of the present disclosure, the third module 230 may include an MLP module, and the MLP module may include n fully connected layers. The fourth data may be input to the first fully connected layer 651 .

[0093] The third module 230 may perform a multiplication operation between the fourth data and the third weight matrix included in the first fully connected layer 651 and an addition operation with the bias included in the first fully connected layer 651 .

[0094] The third module 230 may perform a multiplication operation between the fifth data output from the first fully connected layer 651 and the fourth weight matrix included in the second fully connected layer 652 and an addition operation with the bias included in the second fully connected layer 652 .

[0095] In addition, the third module 230 can obtain the second feature information by performing a multiplication operation between the sixth data output from the n-1th fully connected layer and the fifth weight matrix included in the nth fully connected layer 659 and an addition operation with the bias included in the nth fully connected layer 659.

[0096] The second feature information output from the nth fully connected layer 659 may be input to the second module 220 .

[0097] Figure 7 is a diagram illustrating a method of processing third feature information performed by a third module according to an embodiment of the present disclosure.

[0098] Reference Figure 7 , the third feature information output from the first feature image generator 320 of the first module 210 may be input to the third module 230 .

[0099] According to an embodiment of the present disclosure, the third feature information may include the classification information F including the first image. class and the first feature information F latent A plurality of pieces of intermediate data (a plurality of first to nth pieces of intermediate data) output during a process of generating a first feature image y1 from input data.

[0100] For example, as referenced Figure 5 As described above, a plurality of first to nth intermediate data may be outputted respectively from a plurality of modules 501 included in the first feature image generator 320 .

[0101] According to an embodiment of the present disclosure, the third module 230 may perform downsampling and upsampling 705 on the third feature information. The third module 230 may obtain difference information by performing an element-by-element subtraction operation 707 between the third feature information and data obtained by performing downsampling and upsampling 705 on the third feature information.

[0102] For example, the third module 230 may obtain first sub-data by performing down-sampling and up-sampling 705 on the first intermediate data, and may obtain a difference between the first intermediate data and the first sub-data as first difference information.

[0103] Furthermore, the third module 230 may obtain second sub-data by performing down-sampling and up-sampling 705 on the second intermediate data, and obtain second difference information by performing an element-by-element subtraction operation 707 between the second intermediate data and the second sub-data.

[0104] Furthermore, the third module 230 may obtain nth sub-data by performing down-sampling and up-sampling 705 on the nth intermediate data, and obtain nth difference information by performing an element-by-element subtraction operation 707 between the nth intermediate data and the nth sub-data.

[0105] The third module 230 may obtain fourth feature information by performing convolution operations between the plurality of pieces of first to n-th difference information respectively included in the first to n-th convolution layers (eg, convolution layers 710 , 720 , 730 ) and weights, respectively.

[0106] For example, the third module 230 may obtain the first intermediate output data by performing a convolution operation between the first difference information and the first weight included in the first convolution layer 710 .

[0107] The third module 230 may obtain second intermediate output data by performing a convolution operation between the second difference information and the second weight included in the second convolution layer 720 .

[0108] In addition, the third module 230 may obtain the nth intermediate output data by performing a convolution operation between the nth difference information and the nth weight included in the nth convolution layer 730. According to an embodiment of the present disclosure, the fourth feature information may include a plurality of first to nth intermediate output data.

[0109] The plurality of pieces of first to n-th intermediate output data (fourth feature information) output from the third module 230 may be input to the second module 220 .

[0110] Will refer to Figure 8 The detailed configuration of the second module 220 is described in detail.

[0111] Figure 8 is a diagram illustrating a second module according to an embodiment of the present disclosure.

[0112] Reference Figure 8, the pre-training parameter 810 may be input to the second module 220. For example, the size of the pre-training parameter 810 may be 512×4×4. According to an embodiment of the present disclosure, the pre-training parameter 810 may be a parameter trained together when training the image processing network 103. However, the embodiments of the present disclosure are not limited thereto.

[0113] According to an embodiment of the present disclosure, the second module 220 may obtain the first data by upsampling the input parameter 810 in the upsampling layer 820 .

[0114] The second module 220 may perform an AdaIn operation via an adaptive instance normalization (AdaIn) operator 830 based on the first data and the second feature information input from the third module 230. The AdaIn operation may be defined as shown in Equation 2 below.

[0115] [Equation 2]

[0116] Where μ(.) represents the mean and σ(.) represents the standard deviation. AdaIn(x,y) may refer to an operation of changing the normalized x values ​​to match the distribution of y by using the mean and standard deviation of y after normalizing the x values ​​by using the mean and standard deviation of x. In other words, this may refer to modulating feature x to match the properties of feature y.

[0117] Therefore, the second module 220 may obtain the second data by modulating the first data to match the attribute of the second feature information through the AdaIn operation in the first AdaIn operator 830 .

[0118] The second module 220 may obtain third data by performing a convolution operation between the second data and the weights included in the convolution layer 840 .

[0119] The second module 220 may perform an AdaIn operation via the AdaIn operator 850 based on the third data and the second feature information input from the third module 230. Therefore, the second module 220 may obtain fourth data by modulating the third data via the AdaIn operation to match the attributes of the second feature information.

[0120] The second module 220 may perform a spatial feature transform (SFT) operation via the SFT operator 860 based on the fourth data and the fourth feature information input from the third module 230 .

[0121] The SFT operation may be defined as shown in Equation 3 below.

[0122] [Equation 3]

[0123] Where ⊙ represents an element-by-element multiplication operation, F represents the fourth data, and γ and β are data based on the fourth feature information. For example, the fourth feature information may be adjusted so that the dimension of the fourth data F is doubled, divided by 2, and assigned to γ ​​and β, respectively. Therefore, F, γ, and β may all have the same dimension.

[0124] SFT(F|γ, β) means that the attribute of the fourth data F is modulated by performing affine transformation using data γ and β on the fourth data F. Affine transformation is a spatial linear mapping method that can preserve points, lines, and planes.

[0125] The second module 220 may obtain fifth data by modulating the attribute of the fourth data using the fourth feature information.

[0126] The second module 220 may include a structure in which the modules 801 are repeatedly arranged in series, wherein the module 801 includes an upsampling layer 820, a first AdaIn operator 830, a convolutional layer 840, a second AdaIn operator 850, and an SFT operator 860. However, the embodiments of the present disclosure are not limited thereto.

[0127] In addition, the plurality of first to nth intermediate output data included in the fourth feature information output from the third module 230 may be respectively input to the SFT operator 860 included in each module 801. For example, the third module 230 may include a structure in which n modules 801 are arranged in series, and the plurality of first to nth intermediate output data may be respectively input to the SFT operators 860 respectively included in the n modules 801.

[0128] The second module 220 may generate a second feature image y2 via n modules 801 .

[0129] Fig. 9 is a flowchart illustrating an operation method of an image processing apparatus according to an embodiment of the present disclosure.

[0130] Reference Fig. 9 , at S910 , the method includes the image processing apparatus 100 obtaining classification information and first feature information of a first image by using one or more neural networks.

[0131] The image processing apparatus 100 may obtain the classification information and the first feature information of the first image by using a feature extraction network including one or more convolutional layers.

[0132] For example, the image processing apparatus 100 may obtain the classification information and the first feature information of the first image by using the first convolution layer, the second convolution layer, the third convolution layer, and the addition layer. Figure 4 This is described in detail, so the description already provided above will be omitted here.

[0133] At S920 , the method includes the image processing apparatus 100 generating a first feature image by performing first image processing on the classification information and the first feature information of the first image.

[0134] The image processing device 100 may generate the first feature image by using a feature image generation network including one or more convolution layers. For example, the image processing device 100 may generate the first feature image by using a fourth convolution layer, a first upsampling layer, a fifth convolution layer, a second upsampling layer, a sixth convolution layer, and an addition layer included in the first feature image generator.

[0135] Because it has been referenced Figure 5 This is described in detail, so the description already provided above will be omitted here.

[0136] At S930 , the method includes the image processing apparatus 100 obtaining second feature information by performing second image processing on the classification information and the first feature information of the first image.

[0137] For example, the image processing apparatus 100 may generate the second feature information by using one or more neural networks. The image processing apparatus 100 may generate the second feature information by using a first linear layer, a concatenation layer, a second linear layer, a normalization layer, and a first fully connected layer to an nth fully connected layer. Figure 6 This is described in detail, so the description already provided above will be omitted here.

[0138] At S940 , the method includes: the image processing apparatus 100 obtains fourth feature information by performing third image processing on the third feature information extracted during the first image processing.

[0139] For example, the image processing apparatus 100 may obtain the third feature information in the process of generating the first feature image based on the classification information of the first image and the first feature information. The image processing apparatus 100 may obtain the fourth feature information by performing downsampling, upsampling, element-by-element subtraction, and convolution operations on the third feature information. Figure 7 This is described in detail, so the description already provided above will be omitted here.

[0140] At S950 , the method includes the image processing apparatus 100 generating a second feature image based on the second feature information obtained in operation S930 and the fourth feature information obtained in operation S940 .

[0141] For example, the image processing apparatus 100 may generate the second feature image by performing up-sampling, AdaIn operation, convolution operation, SFT operation, etc. using the second feature information and the fourth feature information. Figure 8This is described in detail, so the description already provided above will be omitted here.

[0142] At S960 , the method includes the image processing apparatus 100 generating a second image based on the first feature image and the second feature image.

[0143] For example, the image processing apparatus 100 may generate the second image by synthesizing the first feature image and the second feature image.

[0144] According to an embodiment of the present disclosure, the second image may have a higher resolution than the first image, and by removing artifacts and noise from the first image, the second image may be an image with improved quality compared to the first image.

[0145] Fig.10 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present disclosure.

[0146] Fig.10 The image processing apparatus 100 may be a computer program product that uses an image processing network ( Figure 2 103) to perform image processing. According to an embodiment of the present disclosure, the image processing network 103 may include one or more neural networks. For example, the image processing network 103 may include a first module 210, a second module 220, and a third module 230 and an image synthesizer 240. However, the embodiments of the present disclosure are not limited thereto.

[0147] Reference Fig.10 According to an embodiment of the present disclosure, the image processing apparatus 100 may include a processor 110 , a memory 120 , and a display 130 .

[0148] According to an embodiment of the present disclosure, the processor 110 may control all operations of the image processing apparatus 100. According to an embodiment of the present disclosure, the processor 110 may execute one or more programs stored in the memory 120.

[0149] According to an embodiment of the present disclosure, the memory 120 may store various pieces of data, programs, or applications for driving and controlling the image processing apparatus 100. The program stored in the memory 120 may include one or more instructions. The program (one or more instructions) or application stored in the memory 120 may be executed by the processor 110.

[0150] According to an embodiment of the present disclosure, the processor 110 may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), or a video processing unit (VPU). Optionally, according to an embodiment of the present disclosure, the processor 110 may be implemented in the form of a system on chip (SoC) integrating at least one of a CPU, a GPU, or a VPU. Optionally, the processor 110 may also include a neural processing unit (NPU).

[0151] According to an embodiment of the present disclosure, the processor 110 may generate a second image by processing the first image using one or more neural networks. For example, the processor 110 may generate a second image by performing denoising for removing noise from the first image while maintaining its fine edges and textures using the image processing network 103. According to an embodiment of the present disclosure, the second image may have a higher quality than the first image. Alternatively, the processor 110 may generate a second image having a higher resolution than the first image by using the image processing network 103.

[0152] According to an embodiment of the present disclosure, the processor 110 may obtain the classification information and the first feature information of the first image by using the first module 210. In this case, the classification information of the first image may include data indicating the probability that the first image is included in each of a plurality of preset categories. The first module 210 may include a feature extraction network, wherein the feature extraction network includes one or more convolutional layers, and the processor 110 may obtain the classification information and the first feature information of the first image by using the feature extraction network. Figure 4 The configuration of the feature extraction network has been described in detail, so its detailed description will be omitted here.

[0153] In addition, the processor 110 may generate a first feature image by performing first image processing on the classification information and the first feature information of the first image using the first module 210. According to an embodiment of the present disclosure, the first feature image may be an image including features of non-high-frequency components (e.g., low-frequency components and medium-frequency components) included in the first image. However, embodiments of the present disclosure are not limited thereto.

[0154] The first module 210 may include a feature image generation network, wherein the feature image generation network includes one or more convolutional layers, and the processor 110 may generate the first feature image by using the feature image generation network. Figure 5 The configuration and operation of the feature image generation network have been described in detail, so its detailed description will be omitted here.

[0155] The processor 110 may also extract third feature information during a process of performing the first image processing by using the feature image generation network.

[0156] For example, the processor 110 may obtain third feature information including a plurality of pieces of first to nth intermediate data during the execution of the first image processing.

[0157] According to an embodiment of the present disclosure, the processor 110 may use the third module 230 to obtain the second feature information by performing the second image processing on the classification information and the first feature information of the first image. Figure 6This is described in detail, so the description already provided above will be omitted here.

[0158] According to an embodiment of the present disclosure, the processor 110 may use the third module 230 to perform third image processing on the third feature information including the plurality of first to nth intermediate data to obtain the fourth feature information. Figure 7 This is described in detail, so the description already provided above will be omitted here.

[0159] According to an embodiment of the present disclosure, the processor 110 may generate a second feature image based on the second feature information and the fourth feature information by using the second module 220 .

[0160] For example, the processor 110 may generate the second feature image by performing upsampling, AdaIn operation, convolution operation, SFT operation, etc. using the second feature information and the fourth feature information. Figure 8 This is described in detail, so the description already provided above will be omitted here.

[0161] According to an embodiment of the present disclosure, the processor 110 may generate the second image by synthesizing the first feature image and the second feature image.

[0162] In addition, according to an embodiment of the present disclosure, the image processing network 103 may be a network trained by a server or an external device. The external device may train the image processing network 103 based on training data. In this case, the training data may include a plurality of data sets including image data containing noise and image data from which noise is removed while retaining edge features or texture features.

[0163] The server or external device may determine parameter values ​​included in kernels used in a plurality of convolutional layers and parameter values ​​included in weight matrices used in linear layers or fully connected layers, respectively, in the image processing network 103. For example, the server or external device may determine the parameter values ​​in a direction that minimizes the difference (missing information) between image data generated by the image processing network 103 and image data (training data) from which noise is removed while retaining edge features.

[0164] According to an embodiment of the present disclosure, the image processing apparatus 100 may receive a trained image processing network 103 from a server or an external device, and store the trained image processing network 103 in the memory 120. For example, according to an embodiment of the present disclosure, the memory 120 may store the structure and parameter values ​​of the image processing network 103, and according to an embodiment of the present disclosure, the processor 110 may generate a second image by removing noise from the first image while retaining edge features using the parameter values ​​stored in the memory 120.

[0165] According to an embodiment of the present disclosure, the display 130 generates a driving signal by converting an image signal, a data signal, an on-screen display (OSD) signal, a control signal, etc. processed by the processor 110. The display 130 may be implemented as a plasma display panel (PDP), a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a flexible display, or a three-dimensional (3D) display. In addition, the display 130 may be formed as a touch screen to be used as an input device and an output device.

[0166] According to an embodiment of the present disclosure, the display 130 may display a second image obtained by performing image processing using the image processing network 103 .

[0167] Fig.10 The block diagram of the image processing device 100 is provided for illustrating an embodiment of the present disclosure. Each component in the block diagram may be integrated, added, or omitted according to the specifications of the image processing device 100 actually implemented. In other words, two or more components may be combined into a single component, or a single component may be divided into two or more components when necessary. In addition, the functions performed by each block are intended to describe an embodiment of the present disclosure, and specific operations or devices related to the functions do not limit the scope of the present disclosure.

[0168] According to an embodiment of the present disclosure, an image processing apparatus may process an image by using one or more neural networks.

[0169] According to an embodiment of the present disclosure, an image processing apparatus may include a memory storing one or more instructions and at least one processor configured to execute the one or more instructions.

[0170] According to an embodiment of the present disclosure, the at least one processor may obtain the classification information and the first feature information of the first image by executing the one or more instructions stored in the memory.

[0171] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may generate a first feature image of the first image by performing first image processing on the classification information and the first feature information.

[0172] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may obtain the second feature information by performing the second image processing on the classification information and the first feature information.

[0173] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may obtain fourth feature information by performing third image processing on the third feature information extracted during the first image processing.

[0174] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may generate a second feature image of the first image based on the second feature information and the fourth feature information.

[0175] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may generate the second image based on the first feature image and the second feature image.

[0176] The first feature image may include features of non-high frequency components included in the first image.

[0177] The second feature image may include features of a high frequency component included in the first image.

[0178] The second image may be of higher quality than the first image.

[0179] The one or more neural networks may include one or more convolutional neural networks (CNNs).

[0180] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may obtain classification information and first feature information of the first image by using one or more CNNs.

[0181] The first image processing may include upsampling the first feature information.

[0182] The first image, the first characteristic image, and the second characteristic image may all have the same size.

[0183] The second image processing may be performed by an MLP module comprising one or more fully connected layers.

[0184] According to an embodiment of the present disclosure, by executing the one or more instructions stored in the memory, the at least one processor may perform a multiplication operation between the input data fed to the fully connected layer and the weight matrix included in the fully connected layer, and an addition operation with the bias included in the fully connected layer.

[0185] By executing the one or more instructions stored in the memory, the at least one processor may obtain the sub-feature information by performing downscaling and upscaling on the third feature information.

[0186] The at least one processor may obtain difference information between the sub-feature information and the third feature information by executing the one or more instructions stored in the memory.

[0187] By executing the one or more instructions stored in the memory, the at least one processor may generate fourth feature information by performing a convolution operation between the difference information and the pre-trained weights.

[0188] By executing the one or more instructions stored in the memory, the at least one processor may obtain fifth feature information by performing a first operation on the second feature information.

[0189] By executing the one or more instructions stored in the memory, the at least one processor may generate a second feature image by performing a second operation on the fifth feature information and the fourth feature information.

[0190] The first operation may include an AdaIn operation.

[0191] The second operation may include an SFT operation.

[0192] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image using one or more neural networks may include obtaining classification information and first feature information of a first image.

[0193] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include generating a first feature image of a first image by performing first image processing on classification information and first feature information.

[0194] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include obtaining second feature information by performing second image processing on classification information and first feature information.

[0195] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include obtaining fourth feature information by performing third image processing on third feature information extracted during first image processing.

[0196] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include generating a second feature image of a first image based on second feature information and fourth feature information.

[0197] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include generating a second image based on a first feature image and a second feature image.

[0198] The first feature image may include features of non-high frequency components included in the first image.

[0199] The second feature image may include features of a high frequency component included in the first image.

[0200] The second image may be of higher quality than the first image.

[0201] The one or more neural networks may include one or more CNNs.

[0202] Obtaining the classification information and the first feature information of the first image may include obtaining the classification information and the first feature information of the first image by using one or more CNNs.

[0203] The first image processing may include upsampling the first feature information.

[0204] The first image, the first characteristic image, and the second characteristic image may all have the same size.

[0205] The second image processing may be performed by an MLP module comprising one or more fully connected layers.

[0206] Obtaining the second feature information by performing the second image processing on the classification information and the first feature information may include: performing a multiplication operation between input data fed to the fully connected layer and a weight matrix included in the fully connected layer, and then an addition operation with a bias included in the fully connected layer.

[0207] Obtaining the fourth feature information by performing the third image processing on the third feature information extracted during the first image processing may include obtaining sub-feature information by performing downscaling and upscaling on the third feature information.

[0208] Obtaining the fourth feature information by performing the third image processing on the third feature information extracted during the first image processing may include obtaining difference information between the third feature information and the sub-feature information.

[0209] Obtaining the fourth feature information by performing the third image processing on the third feature information extracted during the first image processing may include generating the fourth feature information by performing a convolution operation between the difference information and a pre-trained weight.

[0210] Generating the second feature image of the first image based on the second feature information and the fourth feature information may include obtaining fifth feature information by performing a first operation on the second feature information.

[0211] Generating the second feature image of the first image based on the second feature information and the fourth feature information may include generating the second feature image by performing a second operation on the fifth feature information and the fourth feature information.

[0212] The first operation may include an AdaIn operation.

[0213] The second operation may include an SFT operation.

[0214] The image processing apparatus according to an embodiment of the present disclosure may generate a high-quality second image by adaptively processing a first image through image processing using classification information of the first image included in various categories and feature information.

[0215] Therefore, compared with the existing image processing technology, the image processing according to the embodiment of the present disclosure can show improved performance. For example, compared with the image processed by the existing image processing technology, the degree of improvement in the quality of the generated second image or the degree of noise removal can be increased.

[0216] The operating method of the image processing device according to the embodiment of the present disclosure can be implemented in the form of program commands that can be executed by various types of computers and can be recorded on a non-transitory computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable recording medium may be designed and configured specifically for the present disclosure, or may be known to and usable by technicians in the field of computer software. Examples of non-transitory computer-readable recording media include magnetic media (such as hard disks, floppy disks, and tapes), optical media (such as compact disk read-only memories (CD-ROMs) and digital versatile disks (DVDs)), magneto-optical media (such as floppy disks), and hardware devices (such as ROMs, random access memories (RAMs), flash memories, etc.) that are specifically configured to store and execute program commands. Examples of program commands include not only machine codes such as those created by compilers, but also high-level language codes that can be executed by computers using interpreters, etc.

[0217] In addition, the image processing apparatus and the operating method of the image processing apparatus according to the embodiments of the present disclosure may be included in a computer program product when provided. The computer program product may be traded between a seller and a buyer as a product.

[0218] A computer program product may include a software program and a computer-readable storage medium in which the software program is stored. For example, a computer program product may include a product (e.g., a downloadable application) in the form of a software program electronically distributed by a manufacturer of an electronic device or through an electronic market (e.g., Google Play Store™ and App Store™). For such electronic distribution, at least a portion of the software program may be stored on a storage medium or may be temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a relay server for temporarily storing software programs.

[0219] In a system consisting of a server and a client device, the computer program product may include the storage medium of the server or the storage medium of the client device. Alternatively, in the presence of a third device (e.g., a smart phone) communicatively connected to the server or the client device, the computer program product may include the storage medium of the third device. Alternatively, the computer program product may include the software program itself, which is sent from the server to the client device or the third device or from the third device to the client device.

[0220] In this case, one of the server, the client device and the third device can execute a computer program product to perform the method according to an embodiment of the present disclosure. Optionally, at least two of the server, the client device and the third device can execute the method according to an embodiment of the present disclosure in a distributed manner.

[0221] For example, a server (e.g., a cloud server, an artificial intelligence server, etc.) may execute a computer program product stored therein to control a client device communicatively connected to the server to perform a method according to an embodiment of the present disclosure.

[0222] Although the embodiments of the present disclosure have been described in detail above, the embodiments of the present disclosure should not be construed as limiting the scope of the present disclosure, and various modifications and improvements made by those skilled in the art based on the basic concept of the present disclosure also fall within the scope of the present disclosure defined by the attached claims and their equivalents.

Claims

1. An image processing device, include: a memory storing at least one instruction; as well as at least one processor configured to execute the at least one instruction to: Obtaining classification information of a first image and first feature information of the first image; Generate a first feature image of the first image by performing first image processing on the classification information and the first feature information; obtaining second feature information by performing second image processing on the classification information and the first feature information; obtaining fourth feature information by performing third image processing on the third feature information extracted during the first image processing; generating a second feature image of the first image based on the second feature information and the fourth feature information; A second image is generated based on the first feature image and the second feature image.

2. The image processing device according to claim 1, in, The first feature image includes features of non-high frequency components in the first image, and The second feature image includes features of high-frequency components in the first image.

3. The image processing device according to claim 1 or claim 2, in, The quality of the second image is higher than the quality of the first image.

4. The image processing device according to any one of claims 1 to 3, in, The at least one processor is further configured to execute the at least one instruction to obtain the classification information and the first feature information by using at least one convolutional neural network.

5. The image processing device according to any one of claims 1 to 4, in, The first image processing includes upsampling the first feature information, and The first image, the first feature image and the second feature image have the same size.

6. The image processing device according to any one of claims 1 to 5, in, The second image processing is performed by a multilayer perceptron (MLP) module comprising at least one fully connected layer, wherein the at least one processor is further configured to execute the at least one instruction to perform a multiplication operation between input data fed to the fully connected layer and a weight matrix in the fully connected layer, and an addition operation with a bias in the fully connected layer.

7. The image processing device according to any one of claims 1 to 6, in, The at least one processor is further configured to execute the at least one instruction to: Obtaining sub-feature information by performing downscaling and upscaling on the third feature information; Obtaining difference information between the sub-feature information and the third feature information; The fourth feature information is generated by performing a convolution operation between the difference information and the pre-trained weight.

8. The image processing device according to any one of claims 1 to 7, in, The at least one processor is further configured to execute the at least one instruction to: Obtain fifth feature information by performing a first operation on the second feature information; The second feature image is generated by performing a second operation on the fifth feature information and the fourth feature information.

9. The image processing device according to claim 8, in, The first operation comprises an adaptive instance normalization (AdaIn) operation, wherein the second operation comprises a spatial feature transform (SFT) operation.

10. A method of operating an image processing device, include: Obtaining classification information of the first image and first feature information of the first image; Generate a first feature image of the first image by performing first image processing on the classification information and the first feature information; Obtaining second feature information by performing second image processing on the classification information and the first feature information; obtaining fourth feature information by performing third image processing on the third feature information extracted during the first image processing; generating a second feature image of the first image based on the second feature information and the fourth feature information; A second image is generated based on the first feature image and the second feature image.

11. The method according to claim 10, in, The first feature image includes features of non-high-frequency components in the first image, wherein the second feature image includes features of high-frequency components in the first image.

12. The method according to claim 10 or claim 11, in, The quality of the second image is higher than the quality of the first image.

13. The method according to any one of claims 10 to 12, in, Obtaining the classification information and the first feature information includes: obtaining the classification information and the first feature information of the first image by using at least one convolutional neural network.

14. The method according to any one of claims 10 to 13, in, The third feature information includes: a plurality of pieces of intermediate data output through a process of generating the first feature image.

15. A non-transitory computer-readable recording medium storing computer-readable program codes or instructions, in, The computer readable program code or instructions are executable by a processor to perform a method of image processing, the method comprising: Obtaining classification information of a first image and first feature information of the first image; Generate a first feature image of the first image by performing first image processing on the classification information and the first feature information; Obtaining second feature information by performing second image processing on the classification information and the first feature information; obtaining fourth feature information by performing third image processing on the third feature information extracted during the first image processing; generating a second feature image of the first image based on the second feature information and the fourth feature information; A second image is generated based on the first feature image and the second feature image.