Image processing apparatus and operating method thereof

Abstract

An image processing apparatus includes a memory storing at least one instruction, and at least one processor configured to execute the at least one instruction to: obtain first feature data based on a first image; performing first image processing on the first feature data; obtaining a plurality of pieces of second feature data corresponding to a first region of the first image based on first image processing of the first feature data, wherein the first region comprises a first number of pixels; obtaining third feature data based on the first image; performing second image processing on the third feature data; and obtaining a plurality of pieces of fourth feature data corresponding to a second region of the first image based on the second image processing of the third feature data, the second region including a second number of pixels greater than the first number, and generating a second image based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus for processing an image by using a neural network and a method of the image processing apparatus. Background Art

[0002] With the development of computer technology, data business is growing exponentially, and artificial intelligence has become an important trend leading future innovation. Artificial intelligence is a method of simulating human thinking, so it is infinitely applicable to almost all industries. Representative technologies of artificial intelligence include pattern recognition, machine learning, expert systems, neural networks, and natural language processing.

[0003] Neural networks can imitate the characteristics of human biological nerve cells by using mathematical expressions and using algorithms that simulate human learning ability. Through this algorithm, the neural network is able to generate a mapping between input data and output data, and the ability to generate such a mapping can be expressed as the learning ability of the neural network. In short, the neural network has the generalization ability to generate output data about input data that has not yet been used for learning based on the learning results.

[0004] Neural networks can be used for image processing. Specifically, image processing for removing noise or artifacts of an image or improving the resolution of an image can be performed by using a deep neural network (DNN). Summary of the invention

[0005] Technical solutions to the problem

[0006] Provided is an image processing apparatus that can process an image by using one or more neural networks.

[0007] According to an image processing apparatus for processing an image by using at least one neural network, the image processing apparatus may include: a memory storing at least one instruction; and at least one processor configured to execute the at least one instruction.

[0008] The at least one processor may also be configured to execute at least one instruction to obtain first feature data based on the first image.

[0009] The at least one processor may also be configured to execute at least one instruction to obtain a plurality of second feature data corresponding to a first area of ​​the first image by performing a first image processing on the first feature data, the first area including a first number of pixels.

[0010] The at least one processor may also be configured to execute at least one instruction to obtain third feature data based on the first image.

[0011] The at least one processor may also be configured to execute at least one instruction to obtain a plurality of fourth feature data corresponding to a second area of ​​the first image by performing second image processing on the third feature data, the second area including a second number of pixels greater than the first number.

[0012] The at least one processor may also be configured to execute at least one instruction to generate a second image based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

[0013] Each of the first image processing and the second image processing may include performing a self-attention operation.

[0014] At least one processor can also be configured to execute at least one instruction to: obtain multiple second feature data corresponding to the first areas respectively based on information about the peripheral areas of each of the first areas, and obtain multiple fourth feature data corresponding to the second areas respectively based on information about the peripheral areas of each of the second areas.

[0015] The first number is 1, and each of the first regions may include one pixel.

[0016] The at least one processor may also be configured to execute at least one instruction to: obtain multiple query data, multiple key data, and multiple value data included in the first feature data, the multiple query data, the multiple key data, and the multiple value data respectively corresponding to the first area, obtain a weight matrix based on the multiple query data and the multiple key data, and obtain the multiple second feature data based on the multiple value data and the weight matrix.

[0017] The at least one processor may also be configured to execute at least one instruction to: obtain a correlation matrix based on the plurality of query data and the plurality of key data, and obtain a weight matrix by applying a position bias to the correlation matrix based on a size of the first image and a size of an image used to train at least one neural network.

[0018] The at least one processor may also be configured to execute at least one instruction to: transform third feature data of a third area divided into a first number of pixels into data divided into second areas, and obtain multiple fourth feature data by performing second image processing on each of the second areas.

[0019] At least one processor can also be configured to execute at least one instruction to: obtain multiple first query data, multiple first key data and multiple first value data included in the third feature data, the multiple first query data, multiple first key data and multiple first value data respectively correspond to a third area including a first number of pixels, obtain multiple second query data, multiple second key data and multiple second value data corresponding to the second area by grouping the multiple first query data, the multiple first key data and the multiple first value data into groups corresponding to the second area respectively, obtain a weight matrix based on the multiple second query data and the multiple second key data, and obtain the multiple fourth feature data based on the multiple second value data and the weight matrix.

[0020] The third feature data is obtained from the plurality of pieces of second feature data.

[0021] The at least one neural network may include at least one convolutional neural network, and the at least one processor may be further configured to execute at least one instruction to extract first feature data from the first image by using the at least one convolutional neural network.

[0022] The at least one neural network may include at least one convolutional neural network, and the at least one processor may also be configured to execute at least one instruction to: obtain fifth feature data based on multiple second feature data and multiple fourth feature data, and obtain a second image from the fifth feature data by using the at least one convolutional neural network.

[0023] According to an aspect of the present disclosure, an operating method of an image processing apparatus for processing an image by using at least one neural network may include obtaining first feature data based on a first image.

[0024] An operating method of an image processing apparatus for processing an image by using at least one neural network may include obtaining a plurality of pieces of second feature data corresponding to a first region of a first image by performing first image processing on first feature data, the first region including a first number of pixels.

[0025] An operating method of an image processing apparatus for processing an image by using at least one neural network may include obtaining third feature data based on the first image.

[0026] An operating method of an image processing device for processing an image by using at least one neural network may include obtaining a plurality of fourth feature data corresponding to a second area of ​​a first image by performing second image processing on third feature data, the second area including a second number of pixels greater than the first number.

[0027] An operating method of an image processing apparatus for processing an image by using at least one neural network may include generating a second image based on a plurality of pieces of second feature data and a plurality of pieces of fourth feature data.

[0028] Each of the first image processing and the second image processing may include performing a self-attention operation.

[0029] Obtaining multiple pieces of second feature data may include obtaining multiple pieces of second feature data corresponding to the first areas respectively based on information about the peripheral areas of each of the first areas, and obtaining multiple pieces of fourth feature data may include obtaining multiple pieces of fourth feature data corresponding to the second areas respectively based on information about the peripheral areas of each of the second areas.

[0030] The first number is 1, and each of the first regions may include one pixel.

[0031] Obtaining the multiple pieces of second feature data may include: obtaining multiple pieces of query data, multiple pieces of key data, and multiple pieces of value data included in the first feature data, the multiple pieces of query data, the multiple pieces of key data, and the multiple pieces of value data respectively corresponding to the first area; obtaining a weight matrix based on the multiple pieces of query data and the multiple pieces of key data; and obtaining the multiple pieces of second feature data based on the multiple pieces of value data and the weight matrix.

[0032] Obtaining the weight matrix may include: obtaining a correlation matrix based on multiple query data and multiple key data, and obtaining the weight matrix by applying a position deviation to the correlation matrix based on a size of the first image and a size of an image used to train the at least one neural network.

[0033] Obtaining the plurality of pieces of fourth feature data may include: transforming third feature data divided into a third area including a first number of pixels into data divided into second areas; and obtaining the plurality of pieces of fourth feature data by performing the second image processing on each of the second areas.

[0034] Obtaining multiple fourth feature data may include: obtaining multiple first query data, multiple first key data, and multiple first value data included in the third feature data, the multiple first query data, multiple first key data, and multiple first value data respectively corresponding to a third area including a first number of pixels; obtaining multiple second query data, multiple second key data, and multiple second value data corresponding to the second area by grouping the multiple first query data, the multiple first key data, and the multiple first value data into groups corresponding to the second area respectively; obtaining a weight matrix based on the multiple second query data and the multiple second key data; and obtaining the multiple fourth feature data based on the multiple second value data and the weight matrix.

[0035] The third feature data is obtained from the plurality of pieces of second feature data.

[0036] The at least one neural network may include at least one convolutional neural network, and obtaining the first feature data may include extracting the first feature data from the first image by using the at least one convolutional neural network.

[0037] The at least one neural network may include at least one convolutional neural network, and generating the second image may include: obtaining fifth feature data based on multiple second feature data and multiple fourth feature data; and obtaining the second image from the fifth feature data by using the at least one convolutional neural network.

[0038] At least one computer-readable recording medium may store a program for executing the above-described method. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent through the following description in conjunction with the accompanying drawings, in which:

[0040] 1 is a diagram illustrating an operation of an image processing apparatus according to an embodiment of the present disclosure to process an image by using an image processing network;

[0041] FIG2 is a diagram illustrating a second feature extraction network according to an embodiment of the present disclosure;

[0042] FIG3 is a diagram illustrating a transform block according to an embodiment of the present disclosure;

[0043] FIG4 is a diagram illustrating a first transformation layer according to an embodiment of the present disclosure;

[0044] FIG5 is a diagram illustrating a self-attention operation performed in a first self-attention module according to an embodiment of the present disclosure;

[0045] FIG6 is a diagram illustrating a first self-attention module according to an embodiment of the present disclosure;

[0046] FIG7 is a diagram illustrating a multilayer perceptron module according to an embodiment of the present disclosure;

[0047] FIG8 is a diagram illustrating a second transformation layer according to an embodiment of the present disclosure;

[0048] FIG9 is a diagram illustrating a self-attention operation performed in a second self-attention module according to an embodiment of the present disclosure;

[0049] FIG10 is a diagram illustrating a second self-attention module according to an embodiment of the present disclosure;

[0050] FIG11 is a diagram illustrating a second self-attention module according to an embodiment of the present disclosure;

[0051] FIG12 is a diagram showing a transform block according to an embodiment of the present disclosure;

[0052] FIG13 is a diagram illustrating a transform block according to an embodiment of the present disclosure;

[0053] FIG. 14 is a flowchart illustrating a method of an image processing apparatus according to an embodiment of the present disclosure; and

[0054] FIG. 15 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0055] Throughout the disclosure, the expression "at least one of a, b, or c" indicates only a, only b, only c, both a and b, both a and c, both b and c, or all of a, b, and c.

[0056] The terms used herein will be briefly described, and the present disclosure will be described in detail.

[0057] The terms used herein are those common terms currently widely used in the art in view of the functions in the present disclosure, but these terms may vary according to the intention of a person of ordinary skill in the art, precedents, or new technologies in the art. In addition, some of the terms used herein may be arbitrarily selected by the applicant, and in this case, these terms are defined in detail below. Therefore, the specific terms used herein should be defined based on their unique meanings and the entire context of the present disclosure.

[0058] It should be understood that when a certain part "includes" a certain component, the part does not exclude another component, but may further include another component, unless the context clearly indicates otherwise. In addition, terms such as "... unit" or "... module" refer to a unit that performs at least one function or operation, and the unit can be implemented as hardware or software or a combination of hardware and software.

[0059] The present disclosure will now be described more fully with reference to the accompanying drawings so that those of ordinary skill in the art can perform the present disclosure without difficulty. However, the present disclosure may be embodied in many different forms and is not limited to the embodiments of the present disclosure set forth herein. For the sake of clarity, parts not related to the description of the present disclosure are omitted in the accompanying drawings, and the same parts are represented by the same reference numerals throughout the specification.

[0060] FIG. 1 is a diagram illustrating an operation of an image processing apparatus according to an embodiment of the present disclosure to process an image by using an image processing network.

[0061] 1 , the image processing network 103 may receive the first image 101, may process the first image 101, and may generate the second image 102. The first image 101 may include noise or artifacts, and may be a low-resolution image or a low-quality image. The image processing device 100 may generate the second image 102 by performing denoising to remove noise while maintaining fine edges and textures of the first image 101. In addition, the second image 102 may be an image having a higher resolution than the resolution of the first image 101. In addition, the second image 102 may be an image having better image quality than the image quality of the first image 101. However, the present disclosure is not limited thereto.

[0062] The image processing network 103 according to an embodiment of the present disclosure may include one or more neural networks. For example, the image processing network 103 may include a first feature extraction network 200, a second feature extraction network 300, and an image reconstruction network 400. However, the embodiments of the present disclosure are not limited thereto.

[0063] The first feature extraction network 200 that maps from the image space where the first image 101 (input image) exists to the feature space may include one or more neural networks, such as convolutional neural networks (CNNs). In addition, the second feature extraction network 300 may extract features with higher dimensions than the features extracted by the first feature extraction network 200 based on the features extracted by the first feature extraction network 200.

[0064] In addition, the image reconstruction network 400 can obtain the second image 102 (output image) by using high-dimensional features. The image reconstruction network 400 may include one or more neural networks, such as CNN. The image reconstruction network 400 can generate a second image having a higher resolution than the resolution of the first image by performing upsampling. However, the embodiments of the present disclosure are not limited thereto, and the image processing network 103 according to the embodiments of the present disclosure may include various other neural networks.

[0065] An image processing network according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings.

[0066] FIG. 2 is a diagram illustrating a second feature extraction network 300 according to an embodiment of the present disclosure.

[0067] 2 , the second feature extraction network 300 according to an embodiment of the present disclosure may include one or more transformation blocks 310 , a convolution layer 320 , and a summation layer 330 .

[0068] Each of the transform blocks 310 according to an embodiment of the present disclosure can extract second feature data by using information about peripheral pixels adjacent to each of the pixels included in the input first feature data and information about peripheral areas adjacent to each of certain unit areas included in the input first feature data. The transform blocks will be described in detail with reference to FIG. 3 .

[0069] In the convolution layer 320 according to an embodiment of the present disclosure, a convolution operation may be performed between input data (or input information) input to the convolution layer 320 and a kernel included in the convolution layer 320. In one embodiment, the input data input to the convolution layer 320 may be result data output after performing an operation on the transformation block 310. In addition, although the second feature extraction network 300 includes one convolution layer 320 in FIG. 2 , embodiments of the present disclosure are not limited thereto, and the second feature extraction network 300 may include two or more convolution layers.

[0070] In the summation layer 330 according to an embodiment of the present disclosure, an element-by-element summation operation may be performed on the input data x1 input to the second feature extraction network 300 and the output data output from the convolution layer 320. The output data y1 output from the summation layer 330 may be input to the image reconstruction network 400 of FIG. 1 . However, embodiments of the present disclosure are not limited thereto.

[0071] FIG. 3 is a diagram illustrating a transform block 311 according to an embodiment of the present disclosure.

[0072] The transform block 311 of Fig. 3 may be any one of the transform blocks 310 of Fig. 2. Referring to Fig. 3, the transform block 311 according to an embodiment of the present disclosure may include one or more first residual transform blocks and one or more second residual transform blocks.

[0073] The first residual transform block 350 according to an embodiment of the present disclosure may be referred to as an intra residual transformer block aRTB, and the second residual transform block 370 may be referred to as an inter residual transformer block eRTB.

[0074] In addition, the transform block 311 according to an embodiment of the present disclosure may have a structure in which the first residual transform block 350 and the second residual transform block 370 are alternately arranged. However, the embodiment of the present disclosure is not limited thereto, and the transform block 311 may have a structure in which the first residual transform block 350 and the second residual transform block 370 are arranged in parallel.

[0075] The first residual transform block 350 may include one or more first transform layers 351, convolution layers 352, and summation layers 353. The first transform layer will be described in detail with reference to FIG.

[0076] In the convolution layer 352 according to an embodiment of the present disclosure, a convolution operation may be performed between input data (or input information) input to the convolution layer 352 and a kernel included in the convolution layer 352. In an embodiment, the input data input to the convolution layer 352 may be result data output after performing an operation on the first transform layer 351. In addition, although the first residual transform block 350 includes one convolution layer 352 in FIG. 3 , embodiments of the present disclosure are not limited thereto, and the first residual transform block 350 may include two or more convolution layers.

[0077] In the summation layer 353 according to an embodiment of the present disclosure, an element-by-element summation operation may be performed between input data input to the first residual transform block 350 and output data output from the convolution layer 352. The data output from the summation layer 353 may be input to the second residual transform block 370. However, embodiments of the present disclosure are not limited thereto.

[0078] 3 , the second residual transform block 370 may include one or more second transform layers 371, convolution layers 372, and summation layers 373. The second transform layer will be described in detail with reference to FIG.

[0079] In the convolution layer 372 according to an embodiment of the present disclosure, a convolution operation may be performed between input data (or input information) input to the convolution layer 372 and a kernel included in the convolution layer 372. In an embodiment, the input data input to the convolution layer 372 may be result data output after performing an operation on the second transform layer 371. In addition, although the second residual transform block 370 includes one convolution layer 372 in FIG. 3 , embodiments of the present disclosure are not limited thereto, and the second residual transform block 370 may include two or more convolution layers.

[0080] In the summation layer 373 according to an embodiment of the present disclosure, an element-by-element summation operation may be performed between input data input to the second residual transform block 370 and output data output from the convolution layer 372. The data output from the summation layer 373 may be an input of a first residual transform block located next to the second residual transform block 370. However, embodiments of the present disclosure are not limited thereto.

[0081] FIG. 4 is a diagram illustrating a first transformation layer 410 according to an embodiment of the present disclosure.

[0082] The first transformation layer 410 of FIG4 may be any one of the first transformation layers 351 of FIG3. Referring to FIG4, the first transformation layer 410 according to an embodiment of the present disclosure may include a first normalization layer 411, a first self-attention module 412, a first summation layer 413, a second normalization layer 414, a multi-layer perceptron (MLP) module 415, and a second summation layer 416.

[0083] According to an embodiment of the present disclosure, the first normalization layer 411 may normalize the input data x2 input to the first transformation layer 410. For example, the first normalization layer 411 may normalize the input data x2 so that the sum of the input data input to the first transformation layer 410 is 1. However, the present disclosure is not limited thereto. In one embodiment, the first normalization layer 411 may normalize the input data by using any of a variety of normalization methods. The normalized input data may be the input of the first self-attention module 412.

[0084] The first self-attention module 412 according to an embodiment of the present disclosure may perform a self-attention operation on the first feature data (e.g., normalized input data) input to the first self-attention module 412 to obtain a plurality of second feature data corresponding to a first region including a first number of pixels. In an embodiment, the first number may be one pixel.

[0085] The attention operation refers to obtaining the correlation information (for example, similarity information) between the query data Q and the key data K, obtaining the weight based on the correlation information, reflecting the weight in the value data V mapped to the key data K, and performing a weighted sum operation on the value data V reflecting the weight.

[0086] In one embodiment, an attention operation performed based on query data Q, key data K, and value data V obtained from the same input data may be referred to as a self-attention operation.

[0087] The self-attention operation performed in the first self-attention module 412 according to an embodiment of the present disclosure will be described in detail with reference to FIGS. 5 and 6 .

[0088] FIG. 5 is a diagram illustrating a self-attention operation performed in a first self-attention module according to an embodiment of the present disclosure.

[0089] The first self-attention module 412 according to an embodiment of the present disclosure may be referred to as an internal multi-head self-attention (MSA) module.

[0090] 5 , in the first self-attention module 412 , query data Q, key data K, and value data V may be obtained based on first input data 510 input to the first self-attention module 412 .

[0091] For example, the size of the first input data 510 may be W×H, and the number of channels may be C. For convenience of explanation, FIG. 5 will be described assuming that the first input data 510 includes one channel (C=1).

[0092] The first self-attention module 412 may perform a self-attention process on pixels included in the first input data 510 in units of a patch having a size of M×M. For example, the first self-attention module 412 may perform a self-attention process on pixels included in one patch in units of M×M. 2 In FIG5 , for the convenience of explanation, the pixels x included in one patch are used to perform self-attention processing. 1 , x 2 , ..., x K To describe the self-attention operation. Here, K is M 2 .

[0093] Referring to FIG. 5 , the pixels x included in one tile may be 1 、x 2 ,……,x K With the first weight matrix W Q1 The multiplication operation between them is used to obtain the corresponding pixels x 1 、x 2 ,……,x K In addition, the query data 521 can be obtained by pixel x 1 、x 2 ,……,x K With the second weight matrix W K1 The multiplication operation between them is used to obtain the corresponding pixels x 1 、x 2 ,……,x K Key data 522. In addition, the pixel x 1 、x 2 ,……,x K With the third weight matrix W V1 The multiplication operation between them is used to obtain the corresponding pixels x 1 、x 2 ,……,x K Multiple value data 523.

[0094] The related data E 530 may be obtained by an element-by-element multiplication operation between the plurality of key data 522 and the plurality of query data 521. For example, the related data E 530 may be calculated by Equation 1.

[0095] [Equation 1]

[0096]

[0097] In equation 1, e ij represents the element at position (i, j) in the related data E 530, qi represents the query data corresponding to the i-th pixel xi in the query data 521, and k jRepresents the key data corresponding to the j-th pixel xj in the key data 522.

[0098] The first self-attention module 412 may obtain the weight data A 540 by applying a softmax function to the relevant data E 530. For example, the weight data A 540 may be calculated by Equation 2.

[0099] [Equation 2]

[0100]

[0101] In equation 2, a ij represents the element at position (i, j) in the weight data A 540, and e ij Represents the element at position (i, j) in the related data E530.

[0102] The first self-attention module 412 can obtain the pixel x by performing weighted summation on the weight data A540 and the plurality of value data 523. 1 , x 2 , ..., x K The corresponding first output data 550. For example, the first output data 550 can be calculated by Equation 3.

[0103] [Equation 3]

[0104]

[0105] In equation 3, y i represents the pixel x in the first output data 550 1 、x 2 , …, x K The number of pixels included in i The corresponding eigenvalues.

[0106] FIG6 is a diagram illustrating a first self-attention module according to an embodiment of the present disclosure.

[0107] 6, the first self-attention module 412 according to an embodiment of the present disclosure may include three linear layers (e.g., a first linear layer 611, a second linear layer 612, and a third linear layer 613) arranged in parallel. The three linear layers (e.g., the first linear layer 611, the second linear layer 612, and the third linear layer 613) may be fully connected layers.

[0108] For example, the query data Q corresponding to the first input data x 510 can be obtained by the first input data x 510 and the first weight matrix W included in the first linear layer 611. Q1In addition, the first input data x 510 and the second weight matrix W included in the second linear layer 612 can be obtained. K1 The key data K is obtained by performing a multiplication operation between the first input data x 510 and the third weight matrix W included in the third linear layer 613. V1 The value data V is obtained by performing multiplication operation between them.

[0109] The first self-attention module 412 can be used to obtain the key data K by applying the transposition function to the query data Q and the data K T The first correlation data E1 is obtained by performing an element-by-element multiplication operation between them.

[0110] The first self-attention module 412 may obtain second relevant data E2 by adding the position deviation B to the first relevant data E1 and applying a window mask.

[0111] The position deviation B can be determined by Equation 4.

[0112] [Equation 4]

[0113]

[0114] Here, B[] may represent a position deviation to be applied to the first correlation data, and d(x i , x j ) can represent pixel x i and pixels x j In addition, B, which is a value determined by training the neural network, train [d(x i , x j )] can be a pre-stored value.

[0115] Reflection padding may be applied to the first input data x 510 input to the first self-attention module 412. Reflection padding refers to an operation of performing padding when the size of the first input data x 510 is a multiple of the tile size, for example, when the width W of the first input data x 510 is not a multiple of the tile size M or the height H of the first input data x 510 is not a multiple of the tile size M, so that the size of the first input data is a multiple of the tile size. For example, when the size (resolution) of the first input data x 510 is 126×127 and the tile size M=8, the first self-attention module 412 may perform reflection padding so that the size (resolution) of the first input data x 510 is 128×128.

[0116] In addition, the first input data x 510 may be shifted by applying a window mask. For example, the first self-attention module 412 may shift the tile segmentation position by applying a window mask. The tiles may be arranged in more various ways by shifting the tile segmentation position, and when a tile including one pixel is arranged in various ways, pixels adjacent to the pixel may be arranged in more various ways.

[0117] The first self-attention module 412 may obtain weight data A by applying a softmax function to the second related data E2.

[0118] The first self-attention module 412 may obtain first output data y by performing weighted summation on the weight data A and the value data V.

[0119] Referring back to FIG4 , the first output data y output from the first self-attention module 412 may be an output to the first summation layer 413. In the first summation layer 413, an element-wise summation operation may be performed between the first output data y output from the first self-attention module 412 and the input data “X2” input to the first transformation layer 410.

[0120] The second normalization layer 414 may normalize the second output data output from the first summation layer 413. The normalized data may be an input of the MLP module 415. The MLP module 415 will be described in detail with reference to FIG.

[0121] FIG. 7 is a diagram illustrating an MLP module according to an embodiment of the present disclosure.

[0122] 7 , the MLP module 415 according to an embodiment of the present disclosure may include a first linear layer 710 , a normalization layer 720 , and a second linear layer 730 .

[0123] In the first linear layer 710, the third output data may be obtained by a multiplication operation between the data input to the first linear layer 710 and the first weight matrix included in the first linear layer 710. The third output data may be an input to the normalization layer 720. In the normalization layer 720, the data input to the normalization layer 720 may be normalized. The normalized data may be an input to the second linear layer 730.

[0124] In the second linear layer 730 , fourth output data may be obtained through a multiplication operation between data input to the second linear layer 730 and a second weight matrix included in the second linear layer 730 .

[0125] Although the MLP module 415 includes two linear layers in FIG. 7 , embodiments of the present disclosure are not limited thereto, and the MLP module 415 may include three or more linear layers.

[0126] 4, the fourth output data obtained in the MLP module 415 may be an output to the second summation layer 416. In the second summation layer 416, an element-by-element summation operation may be performed between the fourth output data output from the MLP module 415 and the second output data output from the first summation layer 413.

[0127] FIG. 8 is a diagram illustrating a second transform layer according to an embodiment of the present invention.

[0128] The second transformation layer 810 of Figure 8 may be any one of the second transformation layers 371 of Figure 3. Referring to Figure 8, the second transformation layer 810 according to an embodiment of the present disclosure may include a first normalization layer 811, a second self-attention module 812, a first summation layer 813, a second normalization layer 814, a multi-layer perceptron (MLP) module 815, and a second summation layer 816.

[0129] The first normalization layer 811 according to an embodiment of the present disclosure may normalize the input data x3 input to the second transformation layer 810. For example, the first normalization layer 811 may normalize the input data so that the sum of the input data input to the second transformation layer 810 is 1. However, the embodiments of the present disclosure are not limited thereto, and the first normalization layer 811 may normalize the input data by using any of a variety of normalization methods. The normalized input data may be an input to the second self-attention module 812.

[0130] The second self-attention module 812 according to an embodiment of the present disclosure may perform a self-attention operation on the third feature data (e.g., normalized input data) input to the second self-attention module 812 to obtain a plurality of third feature data corresponding to a second region including a second number of pixels. In an embodiment, the second number may be greater than the first number described in FIG. 5, the second region may be greater than the first region, and each second region may include a plurality of pixels.

[0131] The self-attention operation performed in the second self-attention module 812 according to an embodiment of the present disclosure will be described in detail with reference to FIGS. 9 and 10 .

[0132] FIG. 9 is a diagram illustrating a self-attention operation performed in a second self-attention module according to an embodiment of the present disclosure.

[0133] The second self-attention module 812 according to an embodiment of the present disclosure may be referred to as a multi-head self-attention (MSA) module.

[0134] 9 , in the second self-attention module 812 , query data Q, key data K, and value data V may be obtained based on second input data 910 input to the second self-attention module 812 .

[0135] For example, the size of the second input data 910 may be W×H, and the number of channels may be C. For ease of explanation, FIG. 9 will be described assuming that the second input data 910 includes one channel (C=1). 2 ), the number of patches included in one channel may be N.

[0136] The second self-attention module 812 according to an embodiment of the present disclosure can obtain 1 , P 2 , ..., P N The corresponding feature information.

[0137] For example, the patch 920 P included in the second input data 910 may be 1 , P 2 ,…,P N With the first weight matrix W Q2 The multiplication operation between them is used to obtain the patches 920 P respectively included in the second input data 910 1 , P 2 ,…,P N Corresponding multiple query data 921.

[0138] In addition, the patch 920 P included in the second input data 910 can be 1 , P 2 ,…,P N With the second weight matrix W K2 920 P and ... 1 , P 2 ,…,P N ).

[0139] In addition, the patch 920 P included in the second input data 910 can be 1 , P 2 ,…,P N With the third weight matrix W V2 920 P and P 921 P 922 P 923 ... 1 , P 2 ,…,P N ).

[0140] The related data E 930 may be obtained through an element-by-element multiplication operation between the key data 922 and the query data 921, which has been described in Equation 1 of FIG. 5, and thus, description of the same elements will be omitted.

[0141] The weight data A 940 may be obtained by applying a softmax function to the correlation data E 930, which has been described in Equation 2 of FIG. 5, and therefore, description of the same elements will be omitted.

[0142] The second self-attention module 812 may obtain second output data 950 by performing weighted sum on the weight data A 940 and the plurality of pieces of value data 923, which has been described in Equation 3 of FIG. 5, and thus, the same description will be omitted.

[0143] FIG. 10 is a diagram illustrating a second self-attention module according to an embodiment of the present disclosure.

[0144] 10 , the second self-attention module 812 according to an embodiment of the present disclosure may include a first shaping layer 1010. In the first shaping layer 1010, the third input data may be obtained by shaping the second input data so that pixels included in the same tile among the pixels included in the second input data 910 are grouped. In an embodiment, the third input data may be divided into the tiles 920 of FIG. 9 .

[0145] The second self-attention module 812 according to an embodiment of the present disclosure may include three linear layers (e.g., a first linear layer 1021, a second linear layer 1022, and a third linear layer 1023) arranged in parallel. The three linear layers (e.g., a first linear layer 1021, a second linear layer 1022, and a third linear layer 1023) may be fully connected layers.

[0146] For example, the third input data and the first weight matrix W included in the first linear layer 1021 may be Q2 The query data Q' corresponding to the third input data is obtained by performing a multiplication operation between the third input data and the second weight matrix W included in the second linear layer 1022. K2 The key data K' is obtained by performing a multiplication operation between the third input data and the third weight matrix W included in the third linear layer 1023. V2 The value data V' is obtained by performing a multiplication operation between them.

[0147] The second self-attention module 812 can be obtained by applying the query data Q' and the data K' obtained by applying the transposition function to the key data K'. T The first correlation data E1' is obtained by performing an element-by-element multiplication operation between them.

[0148] The second self-attention module 812 may obtain second relevant data E2 ′ by adding the position deviation B′ to the first relevant data E1 ′ and applying a window mask.

[0149] The position deviation B' can be determined by Equation 5.

[0150] [Equation 5]

[0151]

[0152] Here, B'[] may represent a position deviation to be applied to the first correlation data, and d(P i , P j ) can represent the patch P i And patch P j In addition, B, which is a value determined by training the neural network, ′train [d(P i , P j )] can be a pre-stored value.

[0153] In addition, the second input data 910 can be shifted by applying a window mask. For example, the second self-attention module 812 can shift the tile segmentation position by applying a window mask. The tiles can be arranged in various ways by shifting the tile segmentation position. Therefore, the tiles adjacent to a tile can be arranged in various ways.

[0154] The second self-attention module 812 may obtain weight data A' by applying a softmax function to the second relevant data E2'.

[0155] The second self-attention module 812 may obtain second output data y′ by performing a weighted sum on the weight data A′ and the value data V′.

[0156] In this case, the second output data y' may be divided into tiles, and the second output data y' may be shaped into third output data y in the second shaping layer 1030. The third output data y may be divided into pixels without being divided into tiles.

[0157] Referring back to FIG8 , the third output data y output from the second self-attention module 812 may be an output to the first summation layer 813. In the first summation layer 813, an element-by-element summation operation may be performed between the third output data y output from the second self-attention module 812 and the input data x3 input to the second transformation layer 810.

[0158] The second normalization layer 814 may normalize the fourth output data output from the first summation layer 813. The normalized data may be an input of the MLP module 815. The MLP module 815 has been described in detail with reference to FIG. 7, and thus the same description will be omitted.

[0159] The fifth output data obtained in the MLP module 815 may be an output to the second summation layer 816. In the second summation layer 816, an element-by-element summation operation may be performed between the fifth output data output from the MLP module 815 and the fourth output data output from the first summation layer 813.

[0160] FIG. 11 is a diagram illustrating a second self-attention module according to an embodiment of the present disclosure.

[0161] 11, the second self-attention module 812 according to an embodiment of the present disclosure may include three linear layers (e.g., a first linear layer 1111, a second linear layer 1112, and a third linear layer 1113) arranged in parallel. The three linear layers (e.g., the first linear layer 1111, the second linear layer 1112, and the third linear layer 1113) may be fully connected layers.

[0162] For example, the query data Q1′ corresponding to the second input data can be obtained by the second input data 910 x and the first weight matrix W included in the first linear layer 1111 Q3 In addition, the second input data and the second weight matrix W included in the second linear layer 1112 can be obtained. K3 The key data K1' is obtained by performing a multiplication operation between the second input data and the third weight matrix W included in the third linear layer 1113. V3 The value data V1' is obtained by performing a multiplication operation between them.

[0163] The second self-attention module 812 according to an embodiment of the present disclosure may include a first shaping layer 1121, a second shaping layer 1122, and a third shaping layer 1123. For example, the first shaping layer 1121 may be connected to the first linear layer 1111, and the second query data Q' may be obtained by shaping the first query data Q1' output from the first linear layer 1111. For example, the second query data Q' may be formed by grouping query values ​​corresponding to pixels included in the same tile from query values ​​corresponding to pixels included in the first query data Q1', respectively. That is, the second query data Q' may be divided into a plurality of query data corresponding to the tiles, respectively.

[0164] In addition, the second shaping layer 1122 may be connected to the second linear layer 1112, and the second key data K' may be obtained by shaping the first key data K1' output from the second linear layer 1112. For example, the second key data K' may be formed by grouping key values ​​corresponding to pixels included in the same tile from key values ​​corresponding to pixels included in the first key data K1', respectively. That is, the second key data K' may be divided into a plurality of key data pieces corresponding to the tiles, respectively.

[0165] In addition, the third shaping layer 1123 may be connected to the third linear layer 1113, and the second value data V' may be obtained by shaping the first value data V1' output from the third linear layer 1113. For example, the second value data V' may be formed by grouping values ​​corresponding to pixels included in the same tile from values ​​corresponding to pixels included in the first value data V1', respectively. That is, the second value data V' may be divided into a plurality of pieces of value data, respectively corresponding to the tiles.

[0166] The second self-attention module 812 can be obtained by applying the second query data Q' and the data K' obtained by applying the transposition function to the second key data K'. T The first phase data E1' is obtained by performing an element-by-element multiplication operation between them.

[0167] The second self-attention module 812 may obtain second relevant data E2 ′ by adding the position deviation B′ to the first relevant data E1 ′ and applying a window mask.

[0168] The second self-attention module 812 may obtain weight data A' by applying a softmax function to the second relevant data E2'.

[0169] The second self-attention module 812 may obtain second output data y′ by performing a weighted sum on the weight value A′ and the value data V′.

[0170] In this case, the second output data y' may be divided into tiles, and the second output data y' may be shaped into the third output data y in the fourth shaping layer 1130. The third output data y may be divided into pixels without being divided into tiles.

[0171] In the second self-attention module 812 of FIG. 10 , the size of the second input data x may be H×W×C, and the size of the reshaped third input data input to the first linear layer 1021, the second linear layer 1022, and the third linear layer 1023 may be N×M 2 ×C. In one embodiment, N is the number of tiles, M 2 is the size of a patch, and C is the number of channels, and H×W is the same as N×M 2 Therefore, the number of parameters of the weight matrix included in each of the first linear layer 1021, the second linear layer 1022, and the third linear layer 1023 may be M 2 C×M 2 C.

[0172] In contrast, in the second self-attention module 812 of FIG. 11 , the size of the second input data input to the first linear layer 1111, the second linear layer 1112, and the third linear layer 1113 may be H×W×C, and thus the number of parameters of the weight matrix included in each of the first linear layer 1111, the second linear layer 1112, and the third linear layer 1113 may be C×C.

[0173] Therefore, as shown in FIG11, the amount of calculation and the number of calculation parameters when the second input data is calculated in the linear layer and then shaped to be grouped into a plurality of pieces of data corresponding to the tile can be less than the amount of calculation and the number of calculation parameters when the second input data is shaped to be grouped into tiles and then calculated in the linear layer as shown in FIG10. Therefore, the second self-attention module of FIG11 can perform image processing by using a smaller amount of calculation and a smaller number of calculation parameters while maintaining the performance of image processing (inter self-attention).

[0174] FIG. 12 is a diagram illustrating a transform block according to an embodiment of the present disclosure.

[0175] The transform block 1200 of FIG. 12 may be any one of the transform blocks 310 of FIG. 2 .

[0176] 12 , a transform block 1200 according to an embodiment of the present disclosure may include a first residual transform block 1210 , a second residual transform block 1220 , a concatenation layer 1230 , a normalization layer 1240 , a linear layer 1250 , and a summation layer 1260 .

[0177] The transform block 1200 according to an embodiment of the present disclosure may have a structure in which a first residual transform block 1210 and a second residual transform block 1220 are connected to each other in parallel.

[0178] The first residual transform block 1210 according to an embodiment of the present disclosure may be referred to as an intra-residual transform block (aRTB), and the second residual transform block 1220 may be referred to as an inter-residual transform block (eRTB). The first residual transform block 1210 may correspond to the first residual transform block 350 of FIG. 3 , and the second residual transform block 1220 may correspond to the second residual transform block 370 of FIG. 3 . Therefore, the first residual transform block 350, the second residual transform block 370, the first transform layer 410 included in the first residual transform block 350, and the second transform layer 810 included in the second residual transform block 370 described with reference to FIGS. 3 to 11 may be similarly applied to the first residual transform block 1210 and the second residual transform block 1220 of FIG. 12 , and therefore, the same description will be omitted.

[0179] The first input data X1 input to the transform block 1200 according to an embodiment of the present disclosure may be an input to the first residual transform block 1210 and the second residual transform block 1220. The first output data output from the first residual transform block 1210 and the second output data output from the second residual transform block 1220 may be connected in series in the concatenation layer 1230. For example, in the concatenation layer 1230, the third output data obtained by connecting the first output data and the second output data input to the concatenation layer 1230 in series in the channel direction may be an output to the normalization layer 1240. The third output data may be an input to the normalization layer 1240 and normalized by the normalization layer 1240, and the normalized fourth output data may be an input to the linear layer 1250.

[0180] In the linear layer 1250, the fifth output data may be obtained by a multiplication operation between the fourth output data input to the linear layer 1250 and the weight matrix included in the linear layer 1250. The fifth output data may be an input to the summation layer 1260. In addition, the first input data X1 input to the transform block 1200 may be an input to the summation layer 1260.

[0181] In the summation layer 1260 , an element-by-element summation operation may be performed on the first input data and the fifth output data input to the summation layer 1260 .

[0182] The transform block 1200 according to an embodiment of the present disclosure may have a structure in which a module 1201 including a first residual transform block 1210, a second residual transform block 1220, a series layer 1230, a normalization layer 1240, a linear layer 1250, and a summation layer 1260 is repeatedly arranged in series. However, the present disclosure is not limited thereto.

[0183] FIG. 13 is a diagram illustrating a transform block according to an embodiment of the present disclosure.

[0184] The transform block 1300 of FIG. 13 may be any one of the transform blocks 310 of FIG. 2 .

[0185] 13 , a transform block 1300 according to an embodiment of the present disclosure may include a first residual transform block 1310, a second residual transform block 1320, a first normalization layer 1321, a second normalization layer 1322, a first linear layer 1331, a second linear layer 1332, a first attention layer 1341, a second attention layer 1342, a third linear layer 1351, a fourth linear layer 1352, a first summation layer 1361, a second summation layer 1362, a concatenation layer 1370, a third normalization layer 1380, a fifth linear layer 1390, and a third summation layer 1395.

[0186] The transform block 1300 according to an embodiment of the present disclosure may have a structure in which a first residual transform block 1310 and a second residual transform block 1320 are connected to each other in parallel.

[0187] The first residual transform block 1310 according to an embodiment of the present disclosure may be referred to as an intra-residual transform block (aRTB), and the second residual transform block 1320 may be referred to as an inter-residual transform block (eRTB). The first residual transform block 1310 may correspond to the first residual transform block 350 of FIG. 3 , and the second residual transform block 1320 may correspond to the second residual transform block 370 of FIG. 3 . Therefore, the first residual transform block 350, the second residual transform block 370, the first transform layer 410 included in the first residual transform block 350, and the second transform layer 810 included in the second residual transform block 370 described with reference to FIGS. 3 to 11 may be similarly applied to the first residual transform block 1310 and the second residual transform block 1320 of FIG. 13 , and the same description will be omitted.

[0188] The first input data X1 input to the transform block 1300 according to an embodiment of the present disclosure may be an input to the first residual transform block 1310 and the second residual transform block 1320. The first output data output from the first residual transform block 1310 may be an input to the first normalization layer 1321 and normalized by the first normalization layer 1321, and the normalized second output data may be an input to the first linear layer 1331.

[0189] In the first linear layer 1331 , the third output data may be obtained through a multiplication operation between the first normalized data input to the first linear layer 1331 and the first weight matrix included in the first linear layer 1331 .

[0190] The third output data may be used as an attention map that enables focusing on the fourth output data output from the second residual transform block 1320. For example, the third output data and the fourth output data may be inputs to the second attention layer 1342, and in the second attention layer 1342, an element-by-element multiplication operation may be performed between the third output data and the fourth output data.

[0191] In addition, the fourth output data output from the second residual transform block 1320 may be input to the second normalization layer 1322 and normalized by the second normalization layer 1322 , and the normalized fifth output data may be input to the second linear layer 1332 .

[0192] In the second linear layer 1332 , sixth output data may be obtained through a multiplication operation between the fifth output data input to the second linear layer 1332 and the second weight matrix included in the second linear layer 1332 .

[0193] The sixth output data may be used as an attention map that enables focusing on the first output data output from the first residual transform block 1310. For example, the first output data and the sixth output data may be inputs to the first attention layer 1341, and in the first attention layer 1341, an element-by-element multiplication operation may be performed between the first output data and the sixth output data.

[0194] The seventh output data output from the first attention layer 1341 may be an input to the third linear layer 1351 .

[0195] In the third linear layer 1351, the eighth output data may be obtained by a multiplication operation between the seventh output data input to the third linear layer 1351 and the third weight matrix included in the third linear layer 1351. The eighth output data may be an input to the first summation layer 1361. In addition, the first output data output from the first residual transform block 1310 may be an input to the first summation layer 1361. In the first summation layer 1361, an element-by-element summation operation may be performed between the eighth output data input to the first summation layer 1361 and the first output data.

[0196] In addition, the ninth output data output from the second attention layer 1342 may be an input of the fourth linear layer 1352 .

[0197] In the fourth linear layer 1352, the tenth output data may be obtained by a multiplication operation between the ninth output data input to the fourth linear layer 1352 and the fourth weight matrix included in the fourth linear layer 1352. The tenth output data may be an input to the second summation layer 1362. In addition, the fourth output data output from the second residual transform block 1320 may be an input to the second summation layer 1362. In the second summation layer 1362, an element-by-element summation operation may be performed between the tenth output data input to the second summation layer 1362 and the fourth output data.

[0198] The 11th output data output from the first summation layer 1361 may be an input to the concatenation layer 1370 .

[0199] The 12th output data output from the second summation layer 1362 may be an input to the concatenation layer 1370. In the concatenation layer 1370, the 13th output data obtained by concatenating the 11th output data and the 12th output data in the channel direction may be an output to the third normalization layer 1380. The 13th output data may be an input to the third normalization layer 1380 and normalized by the third normalization layer 1380, and the normalized 14th output data may be an input to the fifth linear layer 1390.

[0200] In the fifth linear layer 1390, the 15th output data may be obtained by a multiplication operation between the 14th output data input to the fifth linear layer 1390 and the fifth weight matrix included in the fifth linear layer 1390. The 15th output data may be an input to the third summation layer 1395. In addition, the first input data X1 input to the transform block 1300 may be an input to the third summation layer 1395.

[0201] In the third summation layer 1395 , the 16th output data may be obtained by performing an element-by-element sum operation between the 15th output data input to the third summation layer 1395 and the first input data X1 .

[0202] The transform block 1300 according to an embodiment of the present disclosure may have a structure in which the module 1301 includes a first residual transform block 1310, a second residual transform block 1320, a first normalization layer 1321, a second normalization layer 1322, a first linear layer 1331, a second linear layer 1332, a first attention layer 1341, a second attention layer 1342, a third linear layer 1351, a fourth linear layer 1352, a first summation layer 1361, a second summation layer 1362, a series layer 1370, a third normalization layer 1380, and a fifth linear layer 1390. And the third summation layer 1395 is repeatedly arranged in series. However, the present disclosure is not limited thereto.

[0203] FIG. 14 is a flowchart illustrating a method of an image processing apparatus according to an embodiment of the present disclosure.

[0204] In S1410, the image processing apparatus 100 according to an embodiment of the present disclosure may obtain first feature data based on the first image by using one or more neural networks. For example, the image processing apparatus 100 may extract first feature data corresponding to the first image by using one or more convolutional neural networks.

[0205] In S1420 , the image processing apparatus 100 according to the embodiment of the present disclosure may obtain a plurality of pieces of second feature data by performing the first image processing on the first feature data.

[0206] For example, the image processing apparatus 100 may obtain a plurality of pieces of second feature data corresponding to a first region including a first number of pixels in the first feature data. The image processing apparatus 100 may obtain a plurality of pieces of second feature data respectively corresponding to the first regions based on information about the peripheral region of each of the first regions.

[0207] The image processing apparatus 100 may obtain a plurality of pieces of second feature data by performing self-attention on the first feature data.

[0208] The image processing device 100 according to an embodiment of the present disclosure can obtain a plurality of second feature data corresponding to the first feature data by using the first self-attention module 412 shown and described in Figures 5 and 6. The operations performed in the first self-attention module 412 have been described in detail with reference to Figures 5 and 6, and therefore, the same description will be omitted.

[0209] In S1430 , the image processing apparatus 100 according to the embodiment of the present disclosure may obtain third feature data based on the first image.

[0210] For example, the image processing apparatus 100 may extract the third feature data corresponding to the first image by using one or more convolutional neural networks. Alternatively, the image processing apparatus 100 may obtain the third feature data based on the plurality of second feature data obtained in operation S1420. However, the present disclosure is not limited thereto.

[0211] In S1440 , the image processing apparatus 100 according to the embodiment of the present disclosure may obtain a plurality of pieces of fourth feature data by performing the second image processing on the third feature data.

[0212] For example, the image processing apparatus 100 may obtain a plurality of pieces of fourth feature data corresponding to a second region including a second number of pixels greater than the first number in the third feature data. The image processing apparatus 100 may obtain a plurality of pieces of fourth feature data corresponding to the second regions, respectively, based on information about the peripheral region of each of the second regions.

[0213] The image processing device 100 may obtain a plurality of pieces of fourth feature data by performing self-attention on the third feature data.

[0214] The image processing device 100 according to the embodiment of the present disclosure can obtain a plurality of fourth feature data corresponding to the third feature data by using the second self-attention module 812 shown and described in Figures 9 to 11. The operations performed in the second self-attention module 812 have been described in detail with reference to Figures 9 to 11, and therefore, the same description will be omitted.

[0215] In S1450 , the image processing apparatus 100 according to the embodiment of the present disclosure may generate a second image based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

[0216] For example, the plurality of pieces of second feature data may be feature data based on information about peripheral pixels adjacent to each pixel included in the first image, and the plurality of pieces of fourth feature data may be feature data based on information about peripheral regions (patches) adjacent to each of certain unit regions (patches) included in the first image. Therefore, the image processing device 100 according to an embodiment of the present disclosure can generate a second image by using information about adjacent peripheral pixels (local information) and information about adjacent peripheral regions (non-local information).

[0217] In detail, the image processing apparatus 100 according to an embodiment of the present disclosure may extract feature data from the second feature extraction network 300, may receive the extracted feature data, and may obtain a second image by using the image reconstruction network 400. In one embodiment, the second feature extraction network 300 may include a module (e.g., a first self-attention module) for obtaining a plurality of second feature data and a module (e.g., a second self-attention module) for obtaining a plurality of fourth feature data.

[0218] The second image according to an embodiment of the present disclosure may be an image having a higher resolution than that of the first image, and may be an image having better image quality than that of the first image by removing artifacts, noise, etc. from the first image.

[0219] FIG. 15 is a block diagram showing a configuration of an image processing apparatus according to an embodiment of the present disclosure.

[0220] The image processing device 100 of FIG15 may be a device for performing image processing by using an image processing network 103. The image processing network 103 according to an embodiment of the present disclosure may include one or more neural networks. For example, the image processing network 103 may include a first feature extraction network 200, a second feature extraction network 300, and an image reconstruction network 400. However, the present disclosure is not limited thereto.

[0221] 15 , the image processing apparatus 100 may include a processor 110 , a memory 120 , and a display apparatus 130 .

[0222] The processor 110 according to an embodiment of the present disclosure may control the overall operation of the image processing apparatus 100. The processor 110 according to an embodiment of the present disclosure may execute one or more programs stored in the memory 120.

[0223] The memory 120 according to an embodiment of the present disclosure may store various data, programs or applications for driving and controlling the image processing apparatus 100. The program stored in the memory 120 may include one or more instructions. The program (one or more instructions) or application stored in the memory 120 may be executed by the processor 110.

[0224] The processor 110 according to an embodiment of the present disclosure may include at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a video processing unit (VPU). Alternatively, according to an embodiment of the present disclosure, the processor 110 may be implemented as a system on chip (SoC) in which at least one of the CPU, GPU, and VPU is integrated. Alternatively, the processor 110 may also include a neural processing unit (NPU).

[0225] The processor 110 according to an embodiment of the present disclosure may generate a second image by processing the first image using one or more neural networks. For example, the processor 110 may generate the second image by performing denoising using the image processing network 103 to maintain fine edges and textures while removing noise from the first image. Alternatively, the processor 110 may generate a second image having a higher resolution than the first image by using the image processing network 103.

[0226] The processor 110 according to an embodiment of the present disclosure may obtain first feature data of the first image by using the first feature extraction network 200 .

[0227] The processor 110 according to an embodiment of the present disclosure may obtain a plurality of second feature data of the first image by using the second feature extraction network 300. The structure and operation of the second feature extraction network 300 of FIG. 2 have been described in detail with reference to FIGS. 3 to 14 , and thus a detailed description thereof will be omitted.

[0228] In particular, the processor 110 according to an embodiment of the present disclosure can obtain a plurality of feature data based on information about peripheral pixels adjacent to each pixel included in the first image by using the first self-attention module 412 included in the second feature extraction network 300. The structure and operation of the first self-attention module 412 have been described in detail with reference to FIGS. 5 and 6 , and thus a detailed description thereof will be omitted.

[0229] In addition, the processor 110 according to an embodiment of the present disclosure can obtain a plurality of feature data based on information about a peripheral area (patch) adjacent to each of certain unit areas (patch) included in the first image by using the second self-attention module 812 included in the second feature extraction network 300. The structure and operation of the second self-attention module 812 have been described with reference to FIGS. 9 to 11, and thus a detailed description thereof will be omitted.

[0230] In addition, the processor 110 according to an embodiment of the present disclosure may receive the data extracted by the second feature extraction network 300 by using the image reconstruction network 400, and may obtain a second image by using the image reconstruction network 400.

[0231] The image processing network 103 according to an embodiment of the present disclosure may be a network trained by a server or an external device. The external device may train the image processing network 103 based on training data. In one embodiment, the training data may include a plurality of data sets, the plurality of data sets including image data containing noise and image data retaining edge characteristics or texture characteristics while removing noise.

[0232] The server or the external device may determine parameter values ​​included in kernels respectively used in a plurality of convolutional layers included in the image processing network 103 and parameter values ​​included in weight matrices respectively used in linear layers. For example, the server or the external device may determine parameter values ​​to minimize a difference (loss information) between image data generated by the image processing network 103 and image data retaining edge characteristics while removing noise.

[0233] The image processing apparatus 100 according to an embodiment of the present disclosure may receive a trained image processing network 103 from a server or an external device, and may store the image processing network 103 in the memory 120. For example, the memory 120 may store the structure and parameter values ​​of the image processing network 103 according to an embodiment of the present disclosure, and the processor 110 may generate a second image in which noise is removed from the first image and edge characteristics are retained by using the parameter values ​​stored in the memory 120 according to an embodiment of the present disclosure.

[0234] The display device 130 according to an embodiment of the present disclosure generates a driving signal by transforming an image signal, a data signal, an OSD signal, and a control signal processed by the processor 110. The display device 130 may be implemented as a plasma display panel (PDP), a liquid crystal display (LCD), an organic light emitting diode (OLED), or a flexible display, or may be implemented as a three-dimensional (3D) display. In addition, the display device 130 may include a touch screen, and the touch screen may be used as an output device and an input device.

[0235] The display device 130 according to an embodiment of the present disclosure may display the second image on which image processing is performed by using the image processing network 103 .

[0236] The block diagram of the image processing device 100 of Figure 15 is a block diagram of an embodiment of the present disclosure. The components of the block diagram can be integrated, added, or omitted according to the specifications of the image processing device 100 actually implemented. That is, if necessary, two or more components can be combined into one component, or one component can be divided into two or more components. In addition, the functions performed in each frame are intended to describe the embodiments of the present disclosure, and the detailed operations or devices thereof do not limit the scope of the present disclosure.

[0237] The image processing apparatus according to an embodiment of the present disclosure may process an image by using one or more neural networks.

[0238] An image processing apparatus according to an embodiment of the present disclosure may include a memory storing one or more instructions and at least one processor configured to execute the one or more instructions.

[0239] The at least one processor may be configured to execute the one or more instructions to obtain first feature data based on the first image.

[0240] The at least one processor may be configured to execute one or more instructions to obtain a plurality of pieces of second feature data corresponding to a first area including a first number of pixels by performing a first image processing on the first feature data.

[0241] The at least one processor may be configured to execute one or more instructions to obtain third feature data based on the first image.

[0242] The at least one processor may be configured to execute one or more instructions to obtain fourth feature data corresponding to a second area including a second number of pixels greater than the first number by performing second image processing on the third feature data.

[0243] The at least one processor may be configured to execute one or more instructions to generate a second image based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

[0244] Each of the first image processing and the second image processing may include self-attention.

[0245] The at least one processor may also be configured to execute one or more instructions to obtain a plurality of pieces of second feature data respectively corresponding to the first areas based on information about a peripheral area of ​​each of the first areas.

[0246] The at least one processor may also be configured to execute one or more instructions to obtain a plurality of pieces of fourth feature data respectively corresponding to the second areas based on information about a peripheral area of ​​each of the second areas.

[0247] The first number may be 1, and each first region may include one pixel.

[0248] The image processing device according to the embodiment of the present disclosure can generate a second image by processing the first image based on information about peripheral pixels adjacent to each pixel included in the first image and information about peripheral areas adjacent to each of certain unit areas included in the first image. Therefore, the performance of image processing according to the embodiment of the present disclosure can be better than that of existing image processing technology. For example, the degree of improvement in image quality or the degree of noise removal of the generated second image can be higher than the degree of improvement in image quality or the degree of noise removal of the image processed by the existing image processing technology.

[0249] The at least one processor may also be configured to execute one or more instructions to obtain a plurality of query data, a plurality of key data, and a plurality of value data respectively corresponding to the first regions included in the first feature data.

[0250] The at least one processor may also be configured to execute the one or more instructions to obtain a weight matrix based on the plurality of query data and the plurality of key data.

[0251] The at least one processor may also be configured to execute one or more instructions to obtain a plurality of pieces of second feature data based on the plurality of pieces of value data and the weight matrix.

[0252] The at least one processor may also be configured to execute one or more instructions stored in the memory 120 to obtain a correlation matrix based on the plurality of query data and the plurality of key data.

[0253] The at least one processor may be further configured to execute one or more instructions to obtain a weight matrix by applying a positional bias to the correlation matrix based on a size of the first image and a size of an image used to train the one or more neural networks.

[0254] The at least one processor may be further configured to execute one or more instructions to transform (eg, divide) the third feature data (which is divided into the third region including the first number of pixels) to be divided into the second region.

[0255] The at least one processor may be further configured to execute one or more instructions to obtain a plurality of pieces of fourth feature data by performing second image processing on each of the second regions.

[0256] The at least one processor may also be configured to execute one or more instructions to obtain a plurality of first query data, a plurality of first key data, and a plurality of first value data respectively corresponding to a third area including a first number of pixels included in the third feature data.

[0257] The at least one processor may also be configured to execute the one or more instructions to obtain a plurality of second query data, a plurality of second key data, and a plurality of second value data corresponding to the second area by grouping the plurality of first query data, the plurality of first key data, and the plurality of first value data into groups corresponding to the second area respectively.

[0258] The at least one processor may also be configured to execute the one or more instructions to obtain a weight matrix based on the plurality of pieces of second query data and the plurality of pieces of second key data.

[0259] The at least one processor may also be configured to execute one or more instructions to obtain a plurality of fourth feature data based on the plurality of second value data and the weight matrix.

[0260] The third feature data according to an embodiment of the present disclosure may be obtained from a plurality of pieces of second feature data.

[0261] One or more neural networks according to embodiments of the present disclosure may include one or more convolutional neural networks.

[0262] The at least one processor may also be configured to execute one or more instructions to extract first feature data from the first image by using one or more convolutional neural networks.

[0263] The at least one processor may also be configured to execute one or more instructions to obtain fifth feature data based on the plurality of second feature data and the plurality of fourth feature data.

[0264] The at least one processor may also be configured to execute one or more instructions to obtain a second image from the fifth feature data by using one or more convolutional neural networks.

[0265] An operating method of an image processing apparatus for processing an image by using one or more neural networks according to an embodiment of the present disclosure may include obtaining first feature data based on a first image.

[0266] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include: obtaining a plurality of second feature data corresponding to a first area including a first number of pixels by performing first image processing on first feature data.

[0267] An operating method of an image processing apparatus for processing an image by using one or more neural networks according to an embodiment of the present disclosure may include obtaining third feature data based on a first image.

[0268] According to an embodiment of the present disclosure, an operating method of an image processing device for processing an image by using one or more neural networks may include: obtaining a plurality of fourth feature data corresponding to a second area including a second number of pixels greater than the first number by performing second image processing on third feature data.

[0269] An operating method of an image processing apparatus for processing an image by using one or more neural networks according to an embodiment of the present disclosure may include generating a second image based on a plurality of pieces of second feature data and a plurality of pieces of fourth feature data.

[0270] Each of the first image processing and the second image processing according to an embodiment of the present disclosure may include self-attention.

[0271] Obtaining the plurality of pieces of second feature data may include obtaining the plurality of pieces of second feature data respectively corresponding to the first areas based on information about the peripheral area of ​​each of the first areas.

[0272] Obtaining the plurality of pieces of fourth feature data may include obtaining the plurality of pieces of fourth feature data respectively corresponding to the second areas based on information about the peripheral area of ​​each of the second areas.

[0273] The first number may be 1, and each first region may include one pixel.

[0274] Obtaining the plurality of pieces of second feature data may include obtaining a plurality of pieces of query data, a plurality of pieces of key data, and a plurality of pieces of value data respectively corresponding to the first regions included in the first feature data.

[0275] Obtaining the plurality of pieces of second feature data may include obtaining a weight matrix based on the plurality of pieces of query data and the plurality of pieces of key data.

[0276] Obtaining the plurality of pieces of second feature data may include obtaining the plurality of pieces of second feature data based on the plurality of pieces of value data and a weight matrix.

[0277] Obtaining a weight matrix based on the plurality of query data and the plurality of key data may include obtaining a correlation matrix based on the plurality of query data and the plurality of key data, and obtaining a weight matrix by applying a position deviation to the correlation matrix based on a size of the first image and a size of an image used to train one or more neural networks.

[0278] Obtaining the plurality of pieces of fourth feature data may include transforming the third feature data divided into a third area including the first number of pixels into data divided into the second area.

[0279] Obtaining the plurality of pieces of fourth feature data may include obtaining the plurality of pieces of fourth feature data by performing the second image processing on each of the second regions.

[0280] Obtaining the plurality of pieces of fourth feature data may include obtaining a plurality of pieces of first query data, a plurality of pieces of first key data, and a plurality of pieces of first value data respectively corresponding to third regions including the first number of pixels included in the third feature data.

[0281] Obtaining the plurality of pieces of fourth feature data may include obtaining a plurality of pieces of second query data, a plurality of pieces of second key data, and a plurality of pieces of second value data corresponding to the second area by grouping the plurality of pieces of first query data, the plurality of pieces of first key data, and the plurality of pieces of first value data to correspond to the second area respectively.

[0282] Obtaining the plurality of pieces of fourth feature data may include obtaining a weight matrix based on the plurality of pieces of second query data and the plurality of pieces of second key data.

[0283] Obtaining the plurality of pieces of fourth feature data may include obtaining the plurality of pieces of fourth feature data based on the plurality of pieces of second value data and a weight matrix.

[0284] The third feature data may be obtained from a plurality of pieces of second feature data.

[0285] The one or more neural networks may include one or more convolutional neural networks.

[0286] Obtaining the first feature data may include extracting the first feature data from the first image by using one or more convolutional neural networks.

[0287] Generating the second image may include obtaining fifth feature data based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

[0288] Generating the second image may include obtaining the second image from the fifth feature data by using one or more convolutional neural networks.

[0289] The operation method of the image processing device according to the embodiment of the present disclosure can be implemented as a program command that can be executed by various computer devices, and can be recorded on a computer-readable medium. The computer-readable medium may include program commands, data files, data structures, etc., alone or in combination. The program commands recorded on the computer-readable medium may be specially designed and configured for the present disclosure, or may be known to and can be used by ordinary technicians in the field of computer software. Examples of computer-readable recording media include magnetic media (such as hard disks, floppy disks, or tapes), optical media (such as compact disk read-only memories (CD-ROMs) or digital versatile disks (DVDs)), magneto-optical media (such as optical magnetic floppy disks), and hardware devices (such as ROMs, random access memories (RAMs), or flash memory) specially configured to store and execute program commands. Examples of program commands include high-level language codes that can be executed by a computer by using an interpreter, etc., and machine language codes generated by a compiler.

[0290] Furthermore, the image processing apparatus and the operating method of the image processing apparatus according to the embodiments of the present disclosure may be provided in a computer program product. The computer program product is a product that can be purchased between a seller and a buyer.

[0291] The computer program product may include a S / W program and a computer-readable storage medium storing the S / W program. For example, the computer program product may include a computer program that is downloaded by a manufacturer of an electronic device or an electronic market (e.g., Google Play). TM The S / W program type product (e.g., a downloadable application) is electronically distributed through a store or App store. For electronic distribution, at least a portion of the S / W program may be stored in a storage medium or temporarily generated. In an embodiment, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a relay server that temporarily stores the S / W program.

[0292] The computer program product may include a storage medium of a server or a storage medium of a client device in a system including a server and a client device. Alternatively, when there is a third device (e.g., a smart phone) communicating with the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the software program itself sent from the server to the client device or the third device or from the third device to the client device.

[0293] In an embodiment, one of the server, the client device, and the third device may perform the method according to the embodiment of the present disclosure by executing a computer program product. Alternatively, at least two of the server, the client device, and the third device may perform the method according to the embodiment of the present disclosure in a distributed manner by executing a computer program product.

[0294] For example, a server (eg, a cloud server or an artificial intelligence (AI) server) may execute a computer program product stored in the server, and may control a client device communicating with the server to perform a method according to an embodiment of the present disclosure.

[0295] Although the embodiments of the present disclosure have been described in detail above, the scope of the present disclosure is not limited thereto, and various modifications and improvements made by a person of ordinary skill in the art by using the basic concept of the present disclosure defined by the claims are also within the scope of the present disclosure.

Claims

1. An image processing device for processing an image by using at least one neural network, the image processing device include: A memory storing at least one instruction; as well as at least one processor configured to execute the at least one instruction to: obtaining first feature data based on the first image, By performing the first image processing on the first feature data, a plurality of second feature data corresponding to a first area of ​​the first image is obtained, the first area including a first number of pixels, Obtaining third feature data based on the first image, by performing second image processing on the third feature data, obtaining a plurality of fourth feature data corresponding to a second area of ​​the first image, the second area including a second number of pixels greater than the first number, and A second image is generated based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

2. The image processing device according to claim 1, in, Each of the first image processing and the second image processing includes performing a self-attention operation.

3. The image processing device according to claim 1 or claim 2, in, The at least one processor is further configured to execute the at least one instruction to: based on information about a peripheral area of ​​each of the first areas, obtaining the plurality of pieces of second feature data respectively corresponding to the first areas, and The plurality of pieces of fourth feature data respectively corresponding to the second areas are obtained based on information about the peripheral area of ​​each of the second areas.

4. The image processing device according to any one of claims 1 to 3, in, The first number is 1, and each of the first areas includes one pixel.

5. The image processing device according to any one of claims 1 to 4, in, The at least one processor is further configured to execute the at least one instruction to: Based on the first feature data, a plurality of query data, a plurality of key data, and a plurality of value data are obtained, wherein the plurality of query data, the plurality of key data, and the plurality of value data correspond to the first regions respectively; obtaining a weight matrix based on the plurality of query data and the plurality of key data, and Based on the plurality of pieces of value data and the weight matrix, the plurality of pieces of second feature data are obtained.

6. The image processing device according to claim 5, in, The at least one processor is further configured to execute the at least one instruction to: obtaining a correlation matrix based on the plurality of query data and the plurality of key data, and The weight matrix is ​​obtained by applying a positional deviation to the correlation matrix based on a size of the first image and a size of an image used to train the at least one neural network.

7. The image processing device according to any one of claims 1 to 6, in, The at least one processor is further configured to execute the at least one instruction to: transforming the third feature data divided into a third area including the first number of pixels into the third feature data divided into the second area, and The plurality of pieces of fourth feature data are obtained by performing the second image processing on each of the second areas.

8. The image processing device according to any one of claims 1 to 7, in, The at least one processor is further configured to execute the at least one instruction to: obtaining a plurality of first query data, a plurality of first key data, and a plurality of first value data based on the third feature data, wherein the plurality of first query data, the plurality of first key data, and the plurality of first value data respectively correspond to a third area including the first number of pixels, By grouping the plurality of first query data, the plurality of first key data, and the plurality of first value data into groups corresponding to the second region respectively, a plurality of second query data, a plurality of second key data, and a plurality of second value data corresponding to the second region are obtained, obtaining a weight matrix based on the plurality of second query data and the plurality of second key data, and Based on the plurality of pieces of second value data and the weight matrix, the plurality of pieces of fourth feature data are obtained.

9. The image processing device according to any one of claims 1 to 8, in, The third feature data is obtained from the plurality of pieces of second feature data.

10. The image processing device according to any one of claims 1 to 9, in, The at least one neural network comprises at least one convolutional neural network, and Wherein, the at least one processor is further configured to execute the at least one instruction to extract the first feature data from the first image by using the at least one convolutional neural network.

11. The image processing device according to any one of claims 1 to 10, in, The at least one neural network comprises at least one convolutional neural network, and Wherein, the at least one processor is further configured to execute the at least one instruction to: obtaining fifth feature data based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data, and The second image is obtained from the fifth feature data by using the at least one convolutional neural network.

12. A method for operating an image processing device for processing an image by using at least one neural network, the method comprising: include: obtaining first feature data based on the first image; Obtaining a plurality of pieces of second feature data corresponding to a first region of the first image by performing a first image processing on the first feature data, the first region including a first number of pixels; Obtaining third feature data based on the first image; obtaining, by performing second image processing on the third feature data, a plurality of fourth feature data corresponding to a second area of ​​the first image, the second area including a second number of pixels greater than the first number; and A second image is generated based on the plurality of pieces of second feature data and the plurality of pieces of fourth feature data.

13. The operating method according to claim 12, in, Each of the first image processing and the second image processing includes performing a self-attention operation.

14. The operating method according to claim 12 or claim 13, in, Obtaining the plurality of pieces of second feature data comprises: obtaining the plurality of pieces of second feature data respectively corresponding to the first areas based on information about a peripheral area of ​​each of the first areas, and The obtaining of the plurality of fourth feature data includes: obtaining the plurality of fourth feature data respectively corresponding to the second areas based on information about a peripheral area of ​​each of the second areas.

15. At least one computer-readable recording medium storing a program for execution, the program being executed by a processor to perform the operating method according to any one of claims 12 to 14.

Images