Image processing device and operation method thereof

Abstract

An image processing device is provided, which includes a memory storing one or more instructions and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is further configured to execute the one or more instructions to: generate a second image by performing a deconvolution operation on a first image and a kernel including one or more weights, set values ​​of the one or more weights based on the second image, and adjust the values ​​of the one or more weights based on positions of the one or more weights in the kernel.

Description

Technical Field

[0001] Various embodiments of the present invention relate to an image processing apparatus for performing a deconvolution operation and an operating method thereof, and more particularly, to an image processing apparatus for preventing the occurrence of a checkerboard artifact when performing a deconvolution operation and an operating method thereof. Background Art

[0002] With the development of computer technology, data traffic is growing exponentially, and artificial intelligence has become an important trend to promote future innovation. Because artificial intelligence uses methods that imitate human thinking, it can be applied to all industries infinitely. Examples of representative technologies of artificial intelligence include pattern recognition, machine learning, expert systems, neural networks, and natural language processing.

[0003] Neural networks are modeled by mathematical expressions of human biological neuron properties and use algorithms that simulate human learning capabilities. Through this algorithm, the neural network can generate a mapping between input data and output data, and the ability to generate such a mapping can be expressed as the learning ability of the neural network. In addition, the neural network has a generalization capability that enables the generation of correct output data for input data that has not yet been used for learning based on the results of the learning.

[0004] In a convolutional neural network (CNN) or the like, a deconvolution layer can be used to generate an output image whose size is larger than that of an input image. However, when a deconvolution operation is performed by using a deconvolution layer, the degree of overlap of the kernel varies for each position of the output image depending on the size of the stride used in the deconvolution operation and the size of the kernel. Therefore, there is a problem in that a checkerboard artifact appears in the output image. Summary of the invention

[0005] Solution to the problem

[0006] Various embodiments of the present disclosure may provide an image processing apparatus capable of preventing a checkerboard artifact from occurring when performing a deconvolution operation by performing normalization based on the position of a weight included in a kernel used in the deconvolution operation, and an operating method thereof.

[0007] The beneficial effects of disclosure

[0008] The image processing apparatus according to the embodiment can prevent the occurrence of checkerboard artifacts caused by a deconvolution operation.

[0009] The image processing apparatus according to the embodiment may resize (eg, expand) an image by performing a deconvolution operation, and generate a high-quality image by resizing the image according to the deconvolution operation.

[0010] The image processing apparatus according to the embodiment can reduce the amount of operations and the size of memory by adjusting the size of the image using the deconvolution operation, compared with using other operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A method of generating an image by using a deconvolution operation, performed by an image processing apparatus according to an embodiment, is shown.

[0012] Figure 2 is a reference diagram for describing a process of performing a deconvolution operation according to an embodiment.

[0013] Figure 3 is a reference diagram for describing in detail a process of performing a deconvolution operation according to an embodiment.

[0014] Figure 4 is a reference diagram for describing checkerboard artifacts that occur when a deconvolution operation is performed.

[0015] Figure 5 is a flowchart of an operating method of the image processing apparatus according to the embodiment.

[0016] Figure 6 is a reference diagram for describing a method of adjusting values ​​of one or more weights included in a kernel according to an embodiment.

[0017] Figure 7 is a reference diagram for describing a method of dividing weights included in a kernel into a plurality of groups according to an embodiment.

[0018] Figure 8 An image in which a checkerboard artifact occurs and an image in which a checkerboard artifact does not occur according to an embodiment are shown.

[0019] Fig. 9 is a block diagram of a configuration of an image processing apparatus according to an embodiment.

[0020] Fig.10 is a block diagram of a processor according to an embodiment. DETAILED DESCRIPTION

[0021] An image processing device according to an embodiment includes a memory storing one or more instructions and a processor configured to execute the one or more instructions stored in the memory, wherein the processor is further configured to execute the one or more instructions to: generate a second image by performing a deconvolution operation on a first image and a kernel including one or more weights, set values ​​of the one or more weights based on the second image, and adjust the values ​​of the one or more weights based on the position of the one or more weights in the kernel.

[0022] The processor according to the embodiment may be further configured to execute one or more instructions to: divide the one or more weights into a plurality of groups based on positions of the one or more weights, and normalize each of the plurality of groups.

[0023] The processor according to the embodiment may be further configured to execute one or more instructions to: adjust the value of one or more weights so that sums of weights respectively included in a plurality of groups are equal to each other.

[0024] The processor according to the embodiment may be further configured to execute one or more instructions to: adjust the value of the one or more weights so that the sum of the weights included in each of the plurality of groups is 1.

[0025] The processor according to an embodiment may be further configured to execute one or more instructions to: determine the number of the plurality of groups based on a size of the kernel and a size of a stride used in a deconvolution operation.

[0026] The processor according to an embodiment may be further configured to execute one or more instructions to: determine the number of weights included in each of the plurality of groups based on a size of a kernel and a size of a stride used in a deconvolution operation.

[0027] The processor according to an embodiment may be further configured to execute one or more instructions to: adjust the value of the weight by applying a reliability map including a smoothing function to the kernel.

[0028] The smoothing function according to an embodiment may include a function in a form in which a value gradually changes based on the center of the reliability map.

[0029] The smoothing function according to an embodiment may include at least one of a linear function, a Gaussian function, a Laplace function, or a spline function.

[0030] A size of the second image according to an embodiment may be greater than a size of the first image.

[0031] An operating method of an image processing device according to an embodiment includes: generating a second image by performing a deconvolution operation on a first image and a kernel including one or more weights; setting values ​​of the one or more weights based on the second image; and adjusting the values ​​of the one or more weights based on the positions of the one or more weights in the kernel.

[0032] A computer program product according to an embodiment may include one or more computer-readable recording media storing a program for generating a second image by performing a deconvolution operation on a first image and a kernel including one or more weights, setting values ​​of the one or more weights based on a second image, and adjusting the values ​​of the one or more weights based on positions of the one or more weights in the kernel.

[0033] Mode of the present disclosure

[0034] Terms used in the present specification will be briefly described, and the present disclosure will be described in detail.

[0035] Although the terms used in the present disclosure are selected with general terms currently in common use in consideration of the functions in the present disclosure, these terms may change according to the intention of ordinary technicians in the field, judicial precedents, or the introduction of new technologies. In addition, in specific cases, the applicant may actively select terms, and in this case, the meaning of the terms is disclosed in the corresponding description section of the present disclosure. Therefore, the terms used in the present disclosure should not be defined by the simple names of the terms, but by the meaning of the terms and content in the entire disclosure.

[0036] It should be understood that when a part "includes" a component, the part does not exclude another component, but may further include another component, unless the context clearly dictates otherwise. Terms such as "unit" or "module" refer to units that perform at least one function or operation, and these units may be implemented as hardware or software, or as a combination of hardware and software.

[0037] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings to allow those of ordinary skill in the art to easily implement embodiments of the present disclosure. However, the present disclosure may be implemented in various forms and is not limited to the embodiments of the present disclosure described herein. In order to clearly describe the present disclosure, parts not related to the description are omitted in the accompanying drawings, and the same reference numerals represent the same elements throughout the specification.

[0038] Figure 1 A method of generating an image by using a deconvolution operation performed by an image processing apparatus according to an embodiment is shown.

[0039] Reference Figure 1 The image processing apparatus 100 according to the embodiment may generate an image by using the neural network 20. The image processing apparatus 100 may extract feature information about a first image (input) by using the neural network 20, and generate a second image (output) based on the extracted feature information.

[0040] The neural network 20 may include one or more deconvolution layers, and a deconvolution operation 50 may be performed in each deconvolution layer with respect to an image (input) and a kernel input to each deconvolution layer. As a result of the deconvolution operation, an output image (output) may be generated. Figure 2 and Figure 3 The deconvolution operation 50 is described in detail.

[0041] The deconvolution operation 50 may be used to generate an output image whose size is generally larger than the size of an input image in a convolutional neural network (CNN). For example, the deconvolution operation 50 may be used in various fields, such as super-resolution image generation, autoencoders, style transfer, etc. However, the present disclosure is not limited thereto.

[0042] The size of the second image (output) produced as a result of the deconvolution operation is larger than the size of the first image (input).

[0043] When performing deconvolution operations, checkerboard artifacts may appear with a checkerboard shape. Figure 3 and Figure 4 Describe in detail why checkerboard artifacts occur.

[0044] The image processing apparatus 100 according to the embodiment may adjust a weight value included in a kernel for a deconvolution operation so that a checkerboard artifact does not occur.

[0045] Figure 2 is a reference diagram for describing a process of performing a deconvolution operation according to an embodiment.

[0046] For ease of description, Figure 2 , it is assumed that the size of the input data 210 is 2×2, the size of the kernel 230 used in the input data 210 is 3×3, the size of the stride is 2, the size of the output data 250 is 4×4, and the padding value is 1.

[0047] Reference Figure 2 , the image processing apparatus 100 may perform a deconvolution operation by applying the kernel 230 to the upper left pixel 211 of the input data 210. That is, the image processing apparatus 100 may map a value obtained by multiplying the pixel value a by each of the weights w0, w1, w2, w3, w4, w5, w6, w7, and w8 included in the kernel 230 to each pixel included in the first region 261 of the output data 250. In this regard, the image processing apparatus 100 may determine the starting position of the first region 261 by considering that the padding value is 1 (for example, the starting point of the first region 261 may be a point moved one pixel to the left and top from the first pixel 251 of the output data 250).

[0048] The image processing device 100 may map a value a*w4 obtained by multiplying the pixel value a by the weight w4 to the first pixel 251 of the output data 250 , and map a value a*w5 obtained by multiplying the pixel value a by the weight w5 to the second pixel 252 of the output data 250 .

[0049] Furthermore, the image processing apparatus 100 may map a value obtained by multiplying the pixel value b of the upper right pixel 212 of the input data 210 by each of the weights w0 to w8 included in the kernel 230 to each pixel included in the second region 262, which is shifted by two pixels from the first region 261 of the output data 250. For example, the image processing apparatus 100 may map a value b*w3 obtained by multiplying the pixel value b of the input data 210 by the weight w3 to the second pixel 252 of the output data 250, a value b*w4 obtained by multiplying the pixel value b by the weight w4 to the third pixel 253 of the output data 250, and a value b*w5 obtained by multiplying the pixel value b by the weight w5 to the fourth pixel 254 of the output data 250.

[0050] At this point, when the data that is the target of the deconvolution operation is shifted by one pixel in the input data 210, shifting a plurality of pixels of an area (mapping area) to which the result value of the deconvolution operation is mapped in the output data 250 is referred to as a stride. For example, the mapping area may be shifted pixel by pixel, but as Figure 2 As shown, mapping may be performed by moving the mapping area by two pixels or more from the first area 261 to the second area 262. Therefore, the size of the output data (output image) may be determined according to the size of the stride.

[0051] In the same manner, when the target of the deconvolution operation in the input data 210 is scanned pixel by pixel from left to right and from top to bottom, the weights included in the kernel 230 may be multiplied and mapped to the output data 250 .

[0052] Reference Figure 2 , the first area 261 and the second area 262 may overlap each other. In addition, a plurality of values ​​may be respectively mapped to pixels included in the overlapping area, and the pixel value of the output data 250 may be determined as the sum of the values ​​mapped to the pixels. For example, a value a*w5 obtained by multiplying the pixel value a of the input data 210 by the weight w5 and a value b*w3 obtained by multiplying the pixel value b of the input data 210 by the weight w3 may be mapped to the second pixel 252 of the output data 250, and the value of the second pixel 252 may be determined as the sum of a*w5 and b*w3.

[0053] Figure 3 is a reference diagram for describing in detail a process of performing a deconvolution operation according to an embodiment.

[0054] For ease of description, Figure 3In FIG. 3 , it is assumed that the input data 310 , the kernel 320 , and the output data 330 are one-dimensional. In addition, it is assumed that the size of the input data 310 is 5, the size of the kernel 320 used in the input data 310 is 5, the size of the stride is 1, and the size of the output data 330 is 9.

[0055] Reference Figure 3 , by taking the pixel value I of the input data 310 0 The value I obtained by multiplying the weights w0, w1, w2, w3, and w4 included in the kernel 320 0 *w0、I 0 *w1、I 0 *w2、I 0 *w3 and I 0 *w4 may be mapped to first to fifth pixels 331 , 332 , 333 , 334 , and 335 of the output data 330 , respectively.

[0056] In addition, by changing the pixel value I of the input data 310 1 The value I obtained by multiplying the weights w0, w1, w2, w3, and w4 included in the kernel 320 1 *w0、I 1 *w1、I 1 *w2 and I 1 *w3 and I 1 *w4 may be mapped to the second to sixth pixels 332 , 333 , 334 , 335 , and 336 of the output data 330 , respectively.

[0057] By converting the pixel value I of the input data 310 2 The value I obtained by multiplying the weights w0, w1, w2, w3, and w4 included in the kernel 320 2 *w0、I 2 *w1、I 2 *w2、I 2 *w3 and I 2 *w4 may be mapped to third to seventh pixels 333 , 334 , 335 , 336 , and 337 of the output data 330 , respectively.

[0058] By converting the pixel value I of the input data 310 3 The value I obtained by multiplying the weights w0, w1, w2, w3, and w4 included in the kernel 320 3 *w0、I 3 *w1、I 3 *w2、I 3 *w3、I 3 *w4 may be mapped to fourth to eighth pixels 334 , 335 , 336 , 337 , and 338 of the output data 330 , respectively.

[0059] By converting the pixel value I of the input data 310 4 The value I obtained by multiplying the weights w0, w1, w2, w3, and w4 included in the kernel 320 4 *w0、I 4 *w1、I 4 *w2、I 4 *w3、I 4 *w4 may be mapped to fifth to ninth pixels 335 , 336 , 337 , 338 , and 339 of the output data 330 , respectively.

[0060] Therefore, the value O of the first pixel 331 of the output data 330 is 0 isI 0 *w0, the value of the second pixel 332 is O 1 isI 0 *w1+I 1 *w0, the value of the third pixel 333 is O 2 isI 0 *w2+I 1 *w1+I 2 *w0, the value of the fourth pixel 334 is O 3 isI 0 *w3+I 1 *w2+I 2 *w1+I 3 *w0, the value of the fifth pixel 335 is O 4 For I 0 *w4+I 1 *w3+I 2 *w2+I 3 *w1+I 4 *w0.

[0061] When looking at the deconvolution operation based on the input data 310, a pixel value of the input data 310 (eg, I 0 ) is multiplied by each of a plurality of weights (e.g., w0, w1, w2, w3, and w4), and a value 340 obtained by multiplying the plurality of weights is mapped to a plurality of pixels (e.g., 331 to 335) of the output data 330. Therefore, the deconvolution operation corresponds to a scattering operation. In this regard, when the weights (e.g., w0, w1, w2, w3, and w4) included in the kernel 320 change rapidly, a checkerboard artifact may appear in the output data 330. In particular, in a high-frequency region (region with a large pixel value) of the input data 310, when adjacent weights change rapidly, a checkerboard artifact may appear in a region of the output data 330 corresponding to the high-frequency region.

[0062] When looking at the deconvolution operation based on the output data 330, a pixel value (e.g.,4 ) is determined as a value obtained by: dividing a plurality of pixel values ​​of the input data 310 (for example, I 0 ,I 1 ,I 2 ,I 3 and I 4 ) by each of the plurality of weights (eg, w0, w1, w2, w3, and w4) are added together, and thus the deconvolution operation corresponds to a gathering operation.

[0063] In this regard, the weights respectively applied to the pixels included in the output data 330 are not equal to each other. Figure 3 , one weight w0 is applied to the first pixel 331, two weights w0 and w1 are applied to the second pixel 332, three weights w0, w1, and w2 are applied to the third pixel 333, four weights w0, w1, w2, and w3 are applied to the fourth pixel 334, and five weights w0, w1, w2, w3, and w4 are applied to the fifth pixel 335. In this way, when the number of weights respectively applied to pixels included in the output data 330 varies and the weights applied to one pixel are not normalized, the sum of the weights respectively applied to the pixels of the output data 330 may not be constant.

[0064] For example, when the sum of the four weights w0, w1, w2 and w3 applied to the fourth pixel 334 and the sum of the weights w0, w1, w2, w3 and w4 applied to the fifth pixel 335 are not constant, this causes a checkerboard artifact to appear in the output data 330 when a deconvolution operation is performed.

[0065] Figure 4 is a reference diagram for describing checkerboard artifacts that occur when a deconvolution operation is performed.

[0066] Reference Figure 4 , all pixels included in the input data 410 may include the same pixel value (eg, 1). Figure 3 As shown, when a deconvolution operation is performed by applying a kernel including weights that are not normalized to input data 410, output data 420 including checkerboard artifacts may be generated.

[0067] For example, when all pixel values ​​of the input data 410 are "1" (when all input data are "1"), the value of each pixel included in the output data 420 can be expressed as the sum of the weights applied to each pixel. In this regard, when the weight applied to a pixel is not normalized, the sum of the weights respectively applied to the pixels is not a constant. Therefore, Figure 4 As shown, the output data 420 includes a checkerboard artifact having a particular pattern.

[0068] In order to prevent the occurrence of checkerboard artifacts in the output data 420, the image processing apparatus 100 according to the embodiment may adjust the value of the weight so that the sum of the weights respectively applied to the pixels is constant. In addition, the image processing apparatus 100 may adjust the weight so that the sum of the weights applied to each pixel of the output data 420 is "1" so that the value of the pixel of the output data 420 is equal to the value of the pixel of the input data 410 (e.g., "1").

[0069] Therefore, the image processing device 100 according to the embodiment can generate a second image (output image) which is an enlarged image of the first image (input image) by performing a deconvolution operation using a kernel in which the value of the weight is adjusted, and thus checkerboard artifacts may not appear in the second image.

[0070] Figure 5 is a flowchart of an operating method of the image processing apparatus according to the embodiment.

[0071] Reference Figure 5 , the image processing apparatus 100 according to the embodiment may generate a second image by performing a deconvolution operation on the first image and the kernel (operation S510).

[0072] Already referenced Figure 2 and Figure 3 The deconvolution operation is described and thus will not be described in detail.The size of the second image generated as a result of the deconvolution operation is larger than the size of the first image.

[0073] The image processing apparatus 100 according to the embodiment may set the weight value included in the kernel used in the deconvolution operation based on the generated second image (operation S520). For example, the image processing apparatus 100 may set the value of the weight of the kernel by using a training algorithm such as error back propagation or gradient descent.

[0074] The image processing apparatus 100 may compare and analyze the second image generated by the deconvolution operation and the enlarged image of the first image, and may set a value of a weight of a kernel used in the deconvolution operation based on a result of the analysis.

[0075] As reference Figure 3 and Figure 4 As described above, when the values ​​of weights included in the kernel change rapidly, checkerboard artifacts may appear in the output data.

[0076] Furthermore, when weights respectively applied to pixels of output data are not constant (eg, when the number of weights or the sum of weights is not constant), a checkerboard artifact may occur when a deconvolution operation is performed.

[0077] Therefore, the image processing apparatus 100 according to the embodiment may adjust the value of the weight based on the position of the weight included in the kernel (operation S530).

[0078] For example, the image processing apparatus 100 may apply a reliability map to the kernel according to an embodiment so that the value of the weight included in the kernel does not change rapidly. In addition, the image processing apparatus 100 may divide the weights into a plurality of groups based on the positions of the weights included in the kernel, and perform normalization so that the sum of the weights respectively included in the groups is constant (e.g., "1").

[0079] This will refer to Figure 6 and Figure 7 Detailed description.

[0080] Figure 6 is a reference diagram for describing a method of adjusting values ​​of one or more weights included in a kernel according to an embodiment.

[0081] Reference Figure 6 , the image processing apparatus 100 may set the value of one or more weights included in the kernel 610 for the deconvolution operation. In this regard, the value of the weight included in the kernel 610 may be set according to the training and updating of a neural network including a deconvolution layer on which the deconvolution operation is performed. However, the present disclosure is not limited thereto.

[0082] The image processing apparatus 100 may adjust the values ​​of one or more weights included in the kernel 610 by applying the reliability map 620 to the kernel 610 (operation 601). The reliability map 620 according to an embodiment may include a map indicating a smoothing function, and the image processing apparatus 100 may adjust the values ​​of the weights included in the kernel 610 by performing a multiplication operation on the kernel 610 and the reliability map 620. For example, the smoothing function according to an embodiment may include at least one of a linear function, a Gaussian function, a Laplace function, or a spline function, but is not limited thereto. Figure 6 The reliability map 620 shown may be a graph indicating a Gaussian function.

[0083] Furthermore, the smooth function indicated by the reliability map 620 may be a function in which a map center region has a large value and a value thereof becomes smaller away from the map center region, but is not limited thereto.

[0084] According to an embodiment, when the reliability map 620 is applied to the kernel 610, the value of the weight included in the kernel 610 does not change rapidly, and therefore, the checkerboard artifact can be prevented from occurring in the output data. In particular, the checkerboard artifact can be prevented from occurring in a region of the output data corresponding to a high-frequency region (region with a large pixel value) of the input data.

[0085] Back to Figure 6 , the image processing apparatus 100 may group the weights included in the kernel 610 into a plurality of groups 630 based on the positions of the weights in the kernel 610. Figure 7 The method of grouping weights into groups 630 is described in detail.

[0086] The image processing apparatus 100 may normalize the sum of the weights of each group 630. For example, when the first group 631 includes 9 weights and the second group 632 includes 4 weights, the image processing apparatus 100 may normalize the weights so that the sum of the 9 weights included in the first group 631 and the sum of the 4 weights included in the second group 632 are equal to each other. In this regard, the image processing apparatus 100 may normalize the weights so that the sum of the weights included in one group is 1. However, the present disclosure is not limited thereto.

[0087] The image processing apparatus 100 may apply the kernel 640 in which the value of the weight is adjusted to the neural network 20 including the deconvolution layer. Therefore, the image processing apparatus 100 may perform a deconvolution operation by using the kernel in which the value of the weight is adjusted. For example, the image processing apparatus 100 may generate a second image (output image) by performing a deconvolution operation by applying the kernel in which the value of the weight is adjusted to the first image (input image). At this point, the size of the second image is larger than the size of the first image, and a checkerboard artifact does not appear in the second image.

[0088] Figure 7 is a reference diagram for describing a method of dividing weights included in a kernel into a plurality of groups according to an embodiment.

[0089] exist Figure 7 In this embodiment, a method of dividing weights into a plurality of groups when the size (tap) of kernel 710 is 11×11 and the size of the stride is 4 will be described.

[0090] Figure 7 The coordinates 730 shown represent output data, and the horizontal coordinate w represents the horizontal position of a pixel included in the output data, and the vertical coordinate h represents the vertical position of the pixel included in the output data.

[0091] Assuming that the kernel 710 according to the embodiment is represented by a two-dimensional matrix (11×11 matrix), the index indicated on the weight 722 shown at the top of the coordinate 730 indicates the horizontal position j of the weight 722 in the kernel 710. In addition, the index indicated on the weight 721 shown on the left of the coordinate 730 indicates the vertical position i of the weight 721 in the kernel 710.

[0092] Furthermore, considering the size of the stride (eg, an interval of four pixels) and the positions of pixels included in the output data, weights 721 and 722 shown at the top and left of coordinates 730 correspond to the positions of pixels to which the weights are applied.

[0093] For example, the horizontal positions j of the weight applied to the first pixel 731 included in the output data are 1, 5, and 9, and the vertical positions i thereof are 1, 5, and 9. When the horizontal position and the vertical position of the weight are combined, the weight applied to the first pixel 731 is W 1,1 711, W 1,5 715, W 1,9 719, W 5,1 751, W 5,5 755, W 5,9 759, W 9,1 791, W 9,5 795 and W 9,9 799, which are included in the kernel 710.

[0094] In addition, the horizontal position j of the weight applied to the second pixel 732 included in the output data is 3 and 7, and the vertical position i thereof is 3 and 7. When the horizontal position and the vertical position of the weight are combined, the weight applied to the second pixel 732 is w 3,3 、w 3,7 、w 7,3 and w 7,7 , which are included in kernel 710.

[0095] In addition, the horizontal position j of the weight applied to the third pixel 733 included in the output data is 0, 4, and 8, and the vertical position i thereof is 0, 4, and 8. When the horizontal position and the vertical position of the weight are combined, the weight applied to the third pixel 733 is w 0,0 、w 0,4 、w 0,8 、w 4,0 、w 4,4 、w 4,4 、w 8,0 、w 8,4 and w 8,8 , which are included in kernel 710.

[0096] The image processing apparatus 100 may group the weights applied to each pixel included in the output data into groups. Figure 7 As shown, the image processing apparatus 100 may group the nine weights applied to the first pixel 731 into a first group, and represent the first group as a matrix A 0,0In addition, the image processing apparatus 100 may group the four weights applied to the second pixel 732 into a second group, and represent the second group as a matrix A 2,2 , and the nine weights applied to the third pixel 733 can be grouped into a third group, and the third group is represented as a matrix A 3,3 .

[0097] Including in Figure 7 Among the weights in the illustrated kernel 710 , weights illustrated in the same color indicate weights included in the same group (applied to the same pixel).

[0098] When the weights grouped into a group are indicated by a matrix, the matrix size (size (A i,j )) can be expressed as formula 1.

[0099]

[0100] In Formula 1, floor represents a discard operation, s represents the size of the stride, and c can be expressed as Formula 2.

[0101]

[0102] Referring to Formula 1 and Formula 2, the number of multiple groups is determined based on the size (tap) of the kernel and the size of the stride, and the number of weights included in each group is also determined based on the size (tap) of the kernel and the size of the stride.

[0103] Furthermore, the indices of the components included in the matrix A can be expressed as Formula 3.

[0104]

[0105] In formula 3, t M,i It can be expressed as formula 4, and t N,j It can be expressed as formula 5.

[0106] t M，i =(t+1)%s+(M-1)xs

[0107] t N，j =(t+1)%s+(N-1)xs

[0108] In Formula 4 and Formula 5, % represents a remainder operation. For example, (t+1)%s represents the remainder obtained when (t+1) is divided by s.

[0109] For example, when the kernel size (tap) is 11 and the stride size is 4, when applying equations 1 to 5 for calculation, the matrix A 0,0 The size of is 3×3 (M=3, N=3), and the matrix A 0,0The index of the first element is w 9,9 .

[0110] The image processing apparatus 100 according to the embodiment may normalize the sum of component values ​​(weights) included in each matrix with respect to each matrix. For example, the image processing apparatus 100 may adjust the weights so that the sum of the weights included in each matrix is ​​1.

[0111] Figure 8 An image in which a checkerboard artifact occurs and an image in which a checkerboard artifact does not occur according to an embodiment are shown.

[0112] Reference Figure 8 , the image processing apparatus 100 may generate a first output image 821 by performing a deconvolution operation on the input image 810 and the first kernel. In this regard, the first output image 821 may be an image in which a checkerboard artifact has appeared. For example, the first kernel may correspond to Figure 6 The kernel 610 may be a kernel which has not yet executed Figure 6 and Figure 7 A kernel is shown that applies reliability mapping (operation 601) and weight normalization (operation 602).

[0113] On the other hand, the image processing apparatus 100 may generate a second output image 822 by performing a deconvolution operation on the input image 810 and the second kernel. In this regard, the second output image 822 may be an image in which no checkerboard artifacts appear. For example, the second kernel may correspond to Figure 6 The second kernel may be a kernel 640 in which the following Figure 6 and Figure 7 The kernel of applying reliability mapping (operation 601) and weight normalization (operation 602) as described in .

[0114] Fig. 9 is a block diagram of a configuration of an image processing apparatus according to an embodiment.

[0115] Reference Fig. 9 , the image processing apparatus 100 according to an embodiment may include a processor 120 and a memory 130 .

[0116] The processor 120 according to an embodiment may overall control the image processing apparatus 100. The processor 120 according to an embodiment may execute one or more programs stored in the memory 130.

[0117] The memory 130 according to an embodiment may store various data, programs, or applications for driving and controlling the image processing apparatus 100. The program stored in the memory 130 may include one or more instructions. The program (e.g., one or more instructions) or application stored in the memory 130 may be executed by the processor 120.

[0118] The processor 120 according to an embodiment may train the kernel used in the deconvolution operation to generate the second image by performing a deconvolution operation on the first image and the kernel. For example, the processor 120 may set the value of the weight included in the kernel based on the second image. The processor 120 may set the value of the weight of the kernel by using a training algorithm (e.g., error back propagation or gradient descent), but is not limited thereto.

[0119] The processor 120 may adjust the weight based on the position of the weight included in the kernel. For example, the processor 120 may adjust the weight by applying a reliability map to the kernel. In addition, the processor 120 may divide the weight into a plurality of groups based on the position of the weight included in the kernel, and perform normalization so that the sum of the weights respectively included in the groups is constant (for example, is "1"). Figure 6 and Figure 7 This operation is described in detail, so its detailed description will be omitted.

[0120] The processor 120 may generate an output image without checkerboard artifacts by performing a deconvolution operation on the input image using a kernel in which weight values ​​are adjusted. For example, the processor 120 may generate an output image without checkerboard artifacts by applying a kernel to perform a deconvolution operation. Figure 8 The second output image 820, in which the weight values ​​are adjusted to Figure 8 Input image 810.

[0121] Fig.10 is a block diagram of the processor 120 according to an embodiment.

[0122] Reference Fig.10 , the processor 120 according to an embodiment may include a network trainer 1210 , a deconvolution kernel generator 1220 , and an image processor 1230 .

[0123] The network trainer 1210 may train a neural network including a deconvolution layer. In addition, the network trainer 1210 may set a weight value of a kernel used in a deconvolution operation performed on the deconvolution layer. For example, the network trainer 1210 may set a weight value of a kernel used in a deconvolution operation to generate a second image that is an enlarged image of the first image.

[0124] The network trainer 1210 may store the trained neural network or kernel weights in the memory of the image processing apparatus 100. Alternatively, the neural trainer 1210 may store the trained neural network or kernel weights in the memory of a server connected to the image processing apparatus 100 by wire or wirelessly.

[0125] The deconvolution kernel generator 1220 may include a reliability map applier 1221 and a weight normalizer 1222 .

[0126] The reliability map applicator 1221 may apply a reliability map to the kernel trained by the network trainer 1210. The reliability map may include a map indicating a smooth function, and the smooth function may include at least one of a linear function, a Gaussian function, a Laplace function, or a spline function. The reliability map applicator 1221 may adjust the value of the weight included in the trained kernel by performing a multiplication operation on the trained kernel and the reliability map. By performing the reliability map, the value of the weight included in the kernel does not change rapidly, but may change gradually.

[0127] The weight normalizer 1222 may normalize the weights included in the kernel to which the reliability map is applied. For example, the weight normalizer 1222 may divide the weights included in the kernel into a plurality of groups based on the positions of the weights in the kernel. For example, the weight normalizer 1222 may group the weights applied to each pixel included in the output data into groups.

[0128] The weight normalizer 1222 may normalize the sum of the weights of each group. For example, the weight normalizer 1222 may adjust the value of the weight so that the sums of the weights respectively included in the groups are equal to each other (eg, equal to "1").

[0129] The image processor 1230 may perform a deconvolution operation by using a kernel (e.g., a normalized kernel) in which the values ​​of the weights have been adjusted by the reliability map applicator 1221 and the weight normalizer 1222. For example, the image processor 1230 may generate a second image (output image) by performing a deconvolution operation by applying a kernel in which the values ​​of the weights are adjusted to the first image (input image). At this point, the size of the second image is larger than that of the first image, and a checkerboard artifact does not appear in the second image.

[0130] At least one of the network trainer 1210, the deconvolution kernel generator 1220, or the image processor 1230 may be manufactured in the form of a hardware chip and installed on the image processing device 100. For example, at least one of the network trainer 1210, the deconvolution kernel generator 1220, or the image processor 1230 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a dedicated graphics processor (e.g., a CPU) installed on the above-mentioned various image processing devices.

[0131] In this case, the network trainer 1210, the deconvolution kernel generator 1220, and the image processor 1230 may be installed on one image processing device or a separate image processing device, respectively. For example, some of the network trainer 1210, the deconvolution kernel generator 1220, and the image processor 1230 may be included in the image processing device, and the others may be included in the server.

[0132] In addition, at least one of the network trainer 1210, the deconvolution kernel generator 1220, or the image processor 1230 may be implemented as a software module. When at least one of the network trainer 1210, the deconvolution kernel generator 1220, or the image processor 1230 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or an application. Alternatively, a portion of at least one software module may be provided by the OS, and the remaining portion may be provided by an application.

[0133] Fig. 9 and Fig.10 The block diagrams of the image processing device 100 and the processor 120 shown are block diagrams of embodiments of the present disclosure, respectively. The elements of the block diagram can be integrated, added, or omitted according to the specifications of the image processing device 100 actually implemented. That is, when necessary, two or more elements can be integrated into one element, or one element can be divided into two or more elements. The functions performed in each element (or module) are intended to describe the embodiments of the present disclosure, and their detailed operations or devices do not limit the scope of the present disclosure.

[0134] The operating method of the image processing device according to the embodiment can be implemented in the form of a program command, which can be executed by various computer components and recorded in a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc. alone or in combination. The program commands recorded in the computer-readable recording medium may be program commands specially designed and configured for the present embodiment or program commands known to be used by technicians in the field of computer software. Examples of computer-readable recording media may include magnetic media such as hard disks, floppy disks, and tapes; optical media such as compact disk read-only memories (CD-ROMs) and digital versatile disks (DVDs); magneto-optical media such as optical magnetic disks; and hardware devices specially configured to store and execute program commands, such as read-only memories (ROMs), random access memories (RAMs), and flash memories, etc. In addition, examples of program commands include machine language codes created by compilers and high-level language codes that can be executed by computers using interpreters.

[0135] The image processing apparatus and the operating method of the image processing apparatus according to the embodiment may be included in and provided by a computer program product. The computer program product may be traded as a product between a seller and a buyer.

[0136] A computer program product may include a software (S / W) program and a non-transitory computer-readable recording medium in which the S / W program is stored. For example, a computer program product may include a computer program product that is downloaded by a manufacturer or an electronic device or an electronic market (such as Google Play Store). TM or App Store TM ) A product in the form of an S / W program that is electronically distributed (e.g., a downloadable application). For electronic distribution, at least a portion of the S / W program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a server in a manufacturer or an electronic market or a relay server that temporarily stores the S / W program.

[0137] In a system including a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, when there is a third device (e.g., a smart phone) that communicates with the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself, which is transmitted from the server to the client device or the third device or from the third device to the client device.

[0138] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the embodiment of the present disclosure. Alternatively, two or more of the server, the client device, and the third device may execute the computer program product to perform the method according to the embodiment of the present disclosure in a distributed manner.

[0139] For example, a server (eg, a cloud server or an AI server, etc.) may execute a computer program product stored in the server to control a client device communicating with the server to perform a method according to an embodiment of the present disclosure.

[0140] While the present disclosure has been shown and described with reference to certain exemplary embodiments thereof, the scope of the present disclosure is not limited to this description and also includes various modifications and improvements made by one of ordinary skill in the art using the concepts of the present disclosure defined in the appended claims.

Claims

1. An image processing device, comprising: a memory storing one or more instructions; as well as a processor configured to execute the one or more instructions stored in the memory, Wherein, the processor is further configured to execute the one or more instructions to: dividing the one or more weights included in the kernel into a plurality of groups based on a position of the one or more weights in the kernel, a size of the kernel, and a size of a stride used in a deconvolution operation, adjusting the value of the one or more weights so that the sum of the weights respectively included in the plurality of groups is normalized, and A second image is generated by performing the deconvolution operation using the stride on the first image and the kernel with the values ​​of the one or more weights adjusted.

2. The image processing device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to: adjust the values ​​of the one or more weights so that sums of the weights respectively included in the plurality of groups are equal to each other.

3. The image processing device according to claim 2, wherein: The processor is further configured to execute the one or more instructions to: adjust the value of the one or more weights so that a sum of the weights included in each of the plurality of groups is 1.

4. The image processing device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to determine a number of the plurality of groups based on the size of the kernel and the size of the stride used in the deconvolution operation.

5. The image processing device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to determine a number of the weights included in each of the plurality of groups based on the size of the kernel and the size of the stride used in the deconvolution operation.

6. The image processing device according to claim 1, wherein: The processor is further configured to execute the one or more instructions to adjust the value of the weight by applying a reliability map including a smoothing function to the kernel.

7. The image processing device according to claim 6, wherein: The smoothing function includes a function in a form in which a value gradually changes based on the center of the reliability map.

8. The image processing device according to claim 7, wherein: The smoothing function includes at least one of a linear function, a Gaussian function, a Laplace function, and a spline function.

9. The image processing apparatus according to claim 1, wherein: The size of the second image is larger than that of the first image.

10. A method for operating an image processing device, wherein: The operation method comprises: dividing the one or more weights included in the kernel into a plurality of groups based on the position of the one or more weights in the kernel, the size of the kernel, and the size of the stride used in the deconvolution operation, and adjusting the value of the one or more weights so that the sum of the weights respectively included in the plurality of groups is normalized, and A second image is generated by performing the deconvolution operation using the stride on the first image and the kernel with the values ​​of the one or more weights adjusted.

11. The operating method according to claim 10, wherein: Adjusting the values ​​of the one or more weights based on the positions of the one or more weights in the kernel includes adjusting the values ​​of the one or more weights so that sums of weights respectively included in the plurality of groups are equal to each other.

12. The operating method according to claim 11, wherein: Adjusting the values ​​of the one or more weights based on the positions of the one or more weights in the kernel includes adjusting the values ​​of the one or more weights so that a sum of the weights included in each of the plurality of groups is one.

13. The operating method according to claim 10, wherein: Adjusting the values ​​of the one or more weights based on the position of the one or more weights in the kernel includes determining a number of the plurality of groups based on the size of the kernel and the size of the stride used in the deconvolution operation.

Images