Image processing device and operation method thereof

Abstract

An image processing device is provided, comprising: a memory storing at least one instruction; and a processor configured to execute the at least one instruction stored in the memory to perform the following operations: obtain first feature information by performing a convolution operation on a first image and a first kernel included in a first convolution layer among multiple convolution layers; obtain at least one feature information based on the first feature information; obtain second feature information based on the first feature information and the at least one feature information; obtain third feature information by performing a convolution operation on the obtained second feature information and a second kernel included in a second convolution layer, wherein the second convolution layer is a next layer of the first convolution layer among the multiple convolution layers; and obtain an output image based on the third feature information.

Description

Technical Field

[0001] The present disclosure relates to an image processing device for processing an image by using a convolutional neural network and an operating method thereof, and more particularly, to an image processing device capable of improving image processing performance and an operating method thereof. Background Art

[0002] As computer technology develops, data traffic increases exponentially, and artificial intelligence (AI) has become an important trend leading future innovation. Because AI simulates the way people think, it can be applied to virtually all industries. Representative AI technologies include pattern recognition, machine learning, expert systems, neural networks, and natural language processing.

[0003] Neural networks simulate the characteristics of human biological neurons through mathematical expressions and use an algorithm that simulates human learning ability. Through this algorithm, neural networks are able to generate a mapping between input data and output data, and the ability to generate such a mapping can be called the learning ability of the neural network. In addition, neural networks have the generalization ability to generate correct output data for input data that is not used for learning based on the learning results.

[0004] When an image is processed by using a convolutional neural network (CNN), since a convolution operation is performed by using only feature information generated in a previous layer, characteristics that cannot be obtained from the feature information generated in the previous layer cannot be reflected in image processing. Summary of the invention

[0005] Technical Solution

[0006] Provided is an image processing device and an operating method thereof, in which the performance of image processing can be improved by combining feature information obtained from a plurality of characteristic information generated as a result of a convolution operation with a plurality of existing feature information and using the combined feature information for the convolution operation.

[0007] Best Mode

[0008] Provided is an image processing device and an operating method thereof, in which the performance of image processing can be improved by combining feature information obtained from a plurality of characteristic information generated as a result of a convolution operation with a plurality of existing feature information and using the combined feature information for the convolution operation.

[0009] Additional aspects will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the presented embodiments of the disclosure.

[0010] According to an embodiment of the present disclosure, an image processing device includes: a memory storing at least one instruction; and a processor configured to execute the at least one instruction stored in the memory to perform the following operations: obtain first feature information by performing a convolution operation on a first image and a first kernel included in a first convolution layer among multiple convolution layers; obtain at least one feature information based on the first feature information; obtain second feature information based on the first feature information and the at least one feature information; obtain third feature information by performing a convolution operation on the obtained second feature information and a second kernel included in a second convolution layer among the multiple convolution layers, wherein the second convolution layer is a next layer of the first convolution layer; and obtain an output image based on the third feature information.

[0011] The processor may also be configured to execute the at least one instruction to perform the following operations: obtain the at least one characteristic information from the first feature information by performing filtering or conversion on the first feature information.

[0012] The processor may also be configured to execute the at least one instruction to perform the following operations: obtaining frequency characteristic information about the first feature information from the first feature information.

[0013] The processor may also be configured to execute the row of at least one instruction to perform the following operations: obtain first sub-feature information from the at least one piece of characteristic information; and obtain second feature information by combining the first feature information and the first sub-feature information.

[0014] The processor may also be configured to execute the at least one instruction to perform the following operations: obtain first sub-feature information from the at least one piece of feature information by performing a convolution operation on the at least one piece of feature information and the first sub-kernel.

[0015] The processor may also be configured to execute the at least one instruction to perform the following operations: obtain second feature information based on the first feature information, the at least one characteristic information, and the first sub-feature information.

[0016] The processor may also be configured to execute the at least one instruction to perform the following operations: further obtain second sub-feature information from the at least one feature information; and obtain second feature information based on the at least one feature information, the first feature information, the first sub-feature information, and the second sub-feature information.

[0017] The processor may also be configured to execute the at least one instruction to perform the following operations: obtain second sub-feature information from the at least one piece of feature information by performing a convolution operation on the at least one piece of feature information and the second sub-kernel.

[0018] The processor may also be configured to execute the at least one instruction to obtain the second feature information by adding a value obtained by multiplying the first feature information and the first sub-feature information and a value obtained by multiplying the at least one piece of characteristic information and the second sub-feature information.

[0019] According to another embodiment of the present disclosure, an operating method of an image processing device includes: obtaining first feature information by performing a convolution operation on a first image and a first kernel included in a first convolution layer among multiple convolution layers; obtaining at least one feature information based on the first feature information; obtaining second feature information based on the first feature information and the at least one feature information; obtaining third feature information by performing a convolution operation on the obtained second feature information and a second kernel included in a second convolution layer among the multiple convolution layers, wherein the second convolution layer is a next layer of the first convolution layer; and obtaining an output image based on the third feature information.

[0020] According to another embodiment of the present disclosure, an image processing device includes: a memory storing at least one instruction; and a processor configured to execute the at least one instruction stored in the memory to perform the following operations: obtain at least one feature information from an input signal by filtering or transforming the input signal; obtain first feature information of the input signal; obtain first sub-feature information from the at least one feature information by performing a convolution operation on the at least one feature information and a first sub-kernel in a convolution layer among the multiple convolution layers; obtain second sub-feature information from the at least one feature information by performing a convolution operation on the at least one feature information and a second sub-kernel in a convolution layer among the multiple convolution layers; obtain second feature information based on the first feature information, the at least one feature information, the first sub-feature information, and the second sub-feature information; obtain third feature information from the second feature information by performing a convolution operation on the second feature information and a second kernel included in a convolution layer among the multiple convolution layers; and obtain an output image based on the third feature information.

[0021] Beneficial Effects

[0022] The image processing device according to an embodiment of the present disclosure can perform a convolution operation by combining at least one characteristic obtained from existing feature information obtained in CNN with the existing feature information, thereby using other characteristics and features that are not obtainable from the existing feature information in the convolution operation by using CNN.

[0023] At least one piece of characteristic information according to an embodiment of the present disclosure may be intentionally obtained, and by performing a convolution operation after combining the intentionally obtained characteristic information with existing characteristic information, the existing characteristics may be maintained while processing an image in an intended direction.

[0024] The image processing device according to the embodiment of the present disclosure can improve image processing performance without significantly increasing hardware complexity. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a diagram for describing a method of processing an image by using a convolutional neural network, performed by an image processing apparatus according to an embodiment of the present disclosure;

[0026] Figure 2 is a diagram for describing a method of processing an image by using a convolutional neural network, performed by an image processing apparatus according to another embodiment of the present disclosure;

[0027] Figure 3 is a diagram for describing a method for obtaining at least one piece of characteristic information according to an embodiment of the present disclosure;

[0028] Figure 4 is a diagram for describing a method of obtaining at least one piece of sub-feature information from at least one piece of characteristic information, performed by an image processing device according to an embodiment of the present disclosure;

[0029] Figure 5 is a diagram for describing a method of obtaining at least one piece of sub-feature information from at least one piece of characteristic information, performed by an image processing device according to an embodiment of the present disclosure;

[0030] Figure 6 is a diagram for describing a method of obtaining second feature information performed by an image processing device according to an embodiment of the present disclosure;

[0031] Figure 7 is a diagram for describing a method of processing an image by using a convolutional neural network, performed by an image processing apparatus according to another embodiment of the present disclosure;

[0032] Figure 8 is a diagram for describing a method of processing an image by using a convolutional neural network, performed by an image processing apparatus according to another embodiment of the present disclosure;

[0033] Fig. 9 is a diagram for describing a method of processing an image by using a convolutional neural network, performed by an image processing apparatus according to another embodiment of the present disclosure;

[0034] Fig.10 is a flowchart of an operating method of an image processing device according to an embodiment of the present disclosure;

[0035] Fig.11 is a block diagram of a configuration of an image processing apparatus according to an embodiment of the present disclosure; and

[0036] Fig.12is a block diagram of a processor according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0037] Throughout the disclosure, the expression "at least one of a, b, or c" means only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0038] Terms used in the specification will be briefly defined, and embodiments of the present disclosure will be described in detail.

[0039] All terms used herein, including descriptive or technical terms, should be interpreted as having meanings that are obvious to those of ordinary skill in the art. However, these terms may have different meanings according to the intentions of those of ordinary skill in the art, precedents, or the emergence of new technologies. In addition, some terms may be arbitrarily selected by the applicant, and in this case, the meaning of the selected terms will be described in detail in the specific embodiments of the present disclosure. Therefore, the terms used herein must be defined based on the meaning of the terms and the descriptions throughout the specification.

[0040] When a component “includes” or “contains” an element, unless there is a specific description to the contrary, the component may also include other elements without excluding other elements. In the following description, terms such as “unit” and “module” indicate a unit for processing at least one function or operation, wherein the unit and block may be implemented as hardware or software or by combining hardware and software.

[0041] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement the embodiments of the present disclosure. However, embodiments of the present disclosure can be implemented in many different forms and are not limited to those described herein. In the accompanying drawings, components that are not related to the description are omitted in order to clearly describe the present disclosure, and throughout the specification, the same reference numerals in the drawings represent the same or similar elements.

[0042] Figure 1 5 is a diagram for describing a method for processing an image by using a convolutional neural network (CNN) 50, which is performed by an image processing apparatus according to an embodiment of the present disclosure. The method may be performed by Fig.11 The illustrated image processing apparatus 100 which will be described in detail below performs.

[0043] Reference Figure 1 , the image processing apparatus according to an embodiment of the present disclosure can obtain the output image 20 by performing image processing on the input image 10 using the CNN 50.

[0044] Reference Figure 1, CNN 50 may include a plurality of convolutional layers. For example, the plurality of convolutional layers may include a first convolutional layer 51 to an nth convolutional layer 57 (Conv_1 to Conv_n). In addition, the input image input to CNN 50 is a low-resolution image, and the output image 20 may be a high-resolution image, and therefore, CNN 50 may further include an amplifier 58, but the embodiments of the present disclosure are not limited thereto.

[0045] In each of the first to nth convolutional layers 51 to 57, a convolution operation may be performed between a kernel and at least one image (feature map) input to each of the first to nth convolutional layers 51 to 57, and at least one image (or feature map) generated as a result of the convolution operation may be output. In addition, at least one feature map output from the current convolutional layer may be input to the next convolutional layer.

[0046] For example, a value (e.g., feature information) output from the first convolution layer 51 may be input to the second convolution layer 52, and a value output from the second convolution layer 52 may be input to a third convolution layer (not shown).

[0047] By using the CNN 50, the image processing apparatus 100 can obtain a plurality of pieces of feature information about the input image 10, and based on the obtained plurality of pieces of feature information, obtain an output image 20 in which image processing is performed on the input image 10. Here, the image processing may include various types of image processing such as image enhancement, resolution increase, object recognition in an image, detail enhancement, etc., but is not limited thereto.

[0048] The image processing device 100 according to the embodiment of the present disclosure can obtain at least one piece of characteristic information based on the first feature information 60 obtained from the m-th convolution layer 55 in the first convolution layer 51 to the n-th convolution layer 57. For example, the image processing device 100 can filter the first feature information 60 or perform a transformation on the first feature information 60 to obtain at least one piece of characteristic information 70 (e.g., first characteristic information, second characteristic information, and third characteristic information). This will be described in detail below.

[0049] The image processing apparatus 100 may obtain the second feature information based on the at least one feature information 70 and the first feature information 60. For example, the image processing apparatus 100 may obtain the second feature information by combining the first feature information 60 and the at least one feature information 70. Alternatively, the image processing apparatus 100 may obtain at least one sub-feature information from the at least one feature information 70, and obtain the second feature information by combining the first feature information 60 and the at least one sub-feature information. Alternatively, the image processing apparatus 100 may obtain the second feature information by combining the first feature information 60, the at least one feature information 70, and the at least one sub-feature information. This will be described in detail below.

[0050] The obtained second feature information may be input to the m+1th convolutional layer 56, and the input second feature information may be included in the m+1th convolutional layer 56, so that a convolution operation may be performed using the kernel, and an output image 20 may be obtained based thereon.

[0051] Here, the mth convolution layer 55 may include at least one of the first convolution layer 51 to the n-1th convolution layer (not shown), but is not limited thereto. In addition, at least one piece of characteristic information 70 according to an embodiment of the present disclosure may be obtained from the input image 10. For example, the at least one piece of characteristic information 70 may be obtained by filtering the input image 10 or by performing a transformation on the input image 10.

[0052] Figure 2 is a diagram for describing a method of processing an image by using the CNN 50 , which is performed by the image processing apparatus 100 according to another embodiment of the present disclosure.

[0053] Reference Figure 2 According to the embodiment of the present disclosure, the image processing device 100 can obtain at least one piece of characteristic information 330 based on the first feature information 60 obtained from the m-th convolutional layer 55. Figure 3 A method of obtaining at least one piece of characteristic information 330 is described.

[0054] Figure 3 is a diagram for describing a method of obtaining at least one piece of characteristic information according to an embodiment of the present disclosure.

[0055] The image processing apparatus 100 according to an embodiment of the present disclosure may obtain at least one piece of characteristic information by filtering the first feature information 60 obtained from the m-th convolutional layer 55 .

[0056] For example, the image processing apparatus 100 may obtain the frequency characteristic information by performing filtering using a Gaussian filter or performing discrete cosine transform (DCT) or wavelet transform.

[0057] Reference Figure 3, the image processing device 100 can perform a convolution operation on the first feature information 60 by applying four filter kernels (i.e., the first filter kernel to the fourth filter kernel 311, 312, 313 and 314). The image processing device 100 can obtain the first filter information 321 by performing a convolution operation between the first feature information 60 and the first filter kernel 311, obtain the second filter information 322 by performing a convolution operation between the first feature information 60 and the second filter kernel 312, obtain the third filter information 323 by performing a convolution operation between the first feature information 60 and the third filter kernel 313, and obtain the fourth filter information 324 by performing a convolution operation between the first feature information 60 and the fourth filter kernel 314. Figure 3 In FIG. 4 , four filter kernels are shown and described, but the number of filter kernels is not limited thereto. In addition, the filter kernel may be a Gaussian kernel, but is not limited thereto.

[0058] The weight values ​​of the first to fourth filter kernels 311 to 314 for filtering the first feature information 60 may be preset values.

[0059] The image processing device 100 can obtain the first characteristic information 331 by calculating the difference between the first filter information 321 and the second filter information 322, obtain the second characteristic information 332 by calculating the difference between the second filter information 322 and the third filter information 323, and obtain the third characteristic information 333 by calculating the difference between the third filter information 323 and the fourth filter information 324.

[0060] Here, the first to third characteristic information 331 to 333 may be high frequency characteristic information, middle frequency characteristic information, and low frequency characteristic information, respectively.

[0061] Alternatively, the image processing apparatus 100 according to an embodiment of the present disclosure may obtain characteristic information about the line from the first feature information 60 by performing Hough transform, Radon transform, or the like.

[0062] Alternatively, the image processing apparatus 100 may obtain characteristic information about key points from the first feature information 60 by performing Harris corner point, Shi-Tomasi corner point, etc. However, embodiments of the present disclosure are not limited thereto.

[0063] Return to reference Figure 2According to the image processing device 100 of the embodiment of the present disclosure, at least one sub-feature information can be obtained from at least one feature information 330. For example, the first sub-feature information can be obtained via a convolution operation between the at least one feature information 330 and the first sub-kernel included in the first sub-convolution layer 400, and the second sub-feature information can be obtained via a convolution operation between the at least one feature information 330 and the second sub-kernel included in the second sub-convolution layer 500. This will refer to Figure 4 and Figure 5 Give a description.

[0064] Figure 4 and Figure 5 is a diagram for describing a method of obtaining at least one piece of sub-feature information from at least one piece of characteristic information, performed by the image processing apparatus 100 according to an embodiment of the present disclosure.

[0065] Reference Figure 4 , at least one piece of characteristic information according to an embodiment of the present disclosure may include first characteristic information 331 to third characteristic information 333. Here, the first characteristic information 331 to third characteristic information 333 may be obtained from the first feature information 60 according to an embodiment of the present disclosure, and may be high frequency characteristic information, medium frequency characteristic information, and low frequency characteristic information, respectively, but are not limited thereto.

[0066] The image processing apparatus 100 may obtain a plurality of sub-feature information (characteristic feature information) regarding at least one piece of characteristic information by performing a convolution operation between the at least one piece of characteristic information and a sub-kernel.

[0067] For example, the image processing apparatus 100 may i,c1 ) to the third characteristic information 333 (C i,c3 ) performs a convolution operation with the first sub-kernel set 410 to obtain a plurality of first sub-feature information 420. Here, the first sub-kernel set 410 may include three kernels k with a depth n. i,c1,1 ,...,k i,c1,n ,k i,c2,1 ,...,k i,c2,n ,k i,c3,1 ,...,k i,c3,n , and the obtained plurality of first sub-feature information 420(f i,c1,1 ,...,f i,c1,n ,f i,c2,1 ,...,f i,c2,n ,f i,c3,1 ,...,f i,c3,n ) may include three pieces of feature information having a depth n. Here, the depth n may be determined as the same value as the depth of the first feature information obtained from the first convolutional layer, but is not limited thereto.

[0068] In addition, refer to Figure 5 , the image processing device 100 can obtain a plurality of second sub-feature information 520 (f′) by performing a convolution operation between the first characteristic information 331 to the third characteristic information 333 and the second sub-kernel set 510. i,c1,1 ,...,f' i,c1,n ,f' i,c2,1 ,...,f' i,c2,n ,f' i,c3,1 ,...,f' i,c3,n Here, the second sub-kernel set 510 may include three kernels k′ having a depth n. i,c1,1 ,...,k' i,c1,n ,k' i,c2,1 ,...,k' i,c2,n ,k' i,c3,1 ,...,k' i,c3,n, , and the obtained plurality of pieces of second sub-feature information 520 may include three pieces of feature information having a depth n. Here, the depth n may be determined as the same value as the depth of the first feature information obtained from the first convolutional layer, but is not limited thereto.

[0069] The weight values ​​included in the first sub-kernel set 410 and the second sub-kernel set 510 according to an embodiment of the present disclosure may be determined through training, and the weight values ​​of the first sub-kernel set 410 and the second sub-kernel set 510 may be determined to be different values.

[0070] Return to reference Figure 2 , the feature combiner 600 can obtain the second feature information based on the first feature information, at least one piece of feature information, the first sub-feature information and the second sub-feature information.

[0071] This will refer to Figure 6 Give a description.

[0072] Figure 6 2 is a diagram for describing a method of obtaining second feature information performed by the image processing apparatus 100 according to an embodiment of the present disclosure.

[0073] Reference Figure 6 , the feature combiner 600 can be based on the first feature information f obtained from the first convolutional layer i , from the first feature information f i At least one characteristic information C is obtained i,c (C i,c1, C i,c2, C i,c3 ) and from at least one characteristic information C i,c The first sub-feature information f obtained i,c (f i,c1,1,...,f i,c1,n ,f i,c2,1 ,...,f i,c2,n ,f i,c3,1 ,...,f i,c3,n ) and the second sub-feature information f' i,c (f' i,c1,1 ,...,f' i,c1,n ,f' i,c2,1 ,...,f' i,c2,n ,f' i,c3,1 ,...,f' i,c3,n ) to obtain the second feature information f i+1 .

[0074] Here, the multiplication may be performed via element-by-element multiplication, and in order to convert at least one characteristic information C i,c and the second sub-feature information f' i,c Multiply, at least one characteristic information C i,c The depth can be configured with the second sub-feature information f' i,c The depth n of is the same. Furthermore, the addition can be performed via element-wise summation.

[0075] The characteristic combiner 600 can obtain the second characteristic information f by the following operation: i+1 :The first feature information f i and the first sub-feature information f i,c The value obtained by multiplying at least one characteristic information C i,c and the second sub-feature information f' i,c The values ​​obtained by multiplying are added.

[0076] Optionally, the feature combiner 600 may obtain the second feature information f by the following operation: i+1 :The first feature information f i and the first sub-feature information f i,c The value obtained by multiplication is the second sub-feature information f' i,c Add.

[0077] Return to reference Figure 2 , the obtained second feature information can be input into the m+1th convolutional layer 56, the third feature information can be obtained by performing a convolution operation between the input second feature information and the second kernel included in the m+1th convolutional layer 56, and the output image 20 can be obtained based on the third feature information.

[0078] Figure 7 is a diagram for describing a method of processing an image by using the CNN 50 , which is performed by the image processing apparatus 100 according to another embodiment of the present disclosure.

[0079] Reference Figure 7 , the image processing device 100 according to the embodiment of the present disclosure can obtain at least one piece of characteristic information 330 based on the first feature information 60 obtained from the mth convolutional layer 55. Figure 3 A method of obtaining at least one piece of characteristic information is described, so a detailed description thereof is not provided.

[0080] In addition, the image processing apparatus 100 may obtain feature information from the at least one piece of feature information 330. For example, the image processing apparatus 100 may obtain a plurality of pieces of sub-feature information by performing a convolution operation between the at least one piece of feature information 330 and a sub-kernel set.

[0081] The characteristic combiner 600 can combine the first characteristic information f i and the first sub-feature information f i,c Multiply to obtain the second feature information f i+1 .

[0082] Optionally, the feature combiner 600 may combine the first feature information f i and the first sub-feature information f i,c Add to obtain the second feature information f i+1 .

[0083] Here, multiplication may be performed via element-wise multiplication, and addition may be performed via element-wise summation.

[0084] The obtained second feature information f i+1 Input to the m+1th convolutional layer 56, the second feature information f i+1 A convolution operation is performed between the second kernel included in the m+1th convolution layer 56 to obtain the third feature information, and the output image 20 can be obtained based on the third feature information.

[0085] Figure 8 is a diagram for describing a method of processing an image by using the CNN 50 , which is performed by the image processing apparatus 100 according to another embodiment of the present disclosure.

[0086] Reference Figure 8 , the image processing device 100 according to the embodiment of the present disclosure can obtain at least one piece of characteristic information 330 based on the first feature information 60 obtained from the mth convolutional layer 55. Figure 3 A method of obtaining at least one piece of characteristic information is described, so a detailed description thereof is not provided.

[0087] In addition, the image processing apparatus 100 may obtain feature information from the at least one piece of feature information 330. For example, the image processing apparatus 100 may obtain a plurality of pieces of sub-feature information by performing a convolution operation between the at least one piece of feature information 330 and a sub-kernel set.

[0088] The feature combiner 600 can combine the first feature information f i , at least one characteristic information C i,c and the first sub-feature information f i,c To obtain the second feature information f i+1 .

[0089] For example, the characteristic combiner 600 may obtain the second characteristic information f by the following operation: i+1 :Will pass at least one characteristic information C i,c and the first sub-feature information f i,c, The value obtained by multiplying the first feature information f i Add.

[0090] Here, the multiplication may be performed via element-by-element multiplication, and in order to convert at least one characteristic information C i,c With the first sub-feature information f i,c Multiply, at least one characteristic information C i,c The depth can be configured with the first sub-feature information f i,c The depth n of is the same. Furthermore, the addition can be performed via element-wise summation.

[0091] The obtained second feature information f i+1 Input to the m+1th convolutional layer 56, the second feature information f i+1 A convolution operation is performed between the second kernel included in the m+1th convolution layer 56 to obtain the third feature information, and the output image 20 can be obtained based on the third feature information.

[0092] Fig. 9 is a diagram for describing a method of processing an image by using the CNN 50 , which is performed by the image processing apparatus 100 according to another embodiment of the present disclosure.

[0093] Reference Fig. 9 According to the embodiment of the present disclosure, the image processing device 100 can obtain at least one piece of characteristic information 330 from the input image 10. For example, the image processing device 100 can obtain at least one piece of characteristic information 330 by filtering the input image 10 or transforming the input image 10. Figure 3 The method of obtaining at least one piece of characteristic information via filtering is described, so a detailed description thereof is not provided.

[0094] In addition, the image processing device 100 according to the embodiment of the present disclosure can obtain at least one sub-feature information from the at least one feature information 330. For example, the first sub-feature information can be obtained via a convolution operation between the at least one feature information 330 and the first sub-kernel included in the first sub-convolution layer 400, and the second sub-feature information can be obtained via a convolution operation between the at least one feature information 330 and the second sub-kernel included in the second sub-convolution layer 500. Figure 4 and Figure 5 This has been described in detail, so a detailed description thereof will not be provided again.

[0095] The image processing apparatus 100 may obtain the first feature information f from the mth convolutional layer 55 i , and the characteristic combiner 600 can be based on the first feature information f i , at least one characteristic information C i,c , the first sub-feature information f i,c and the second sub-feature information f' i,c Get the second feature information f i+1 .

[0096] The obtained second feature information f i+1 Input to the m+1th convolutional layer 56, the second feature information f i+1 A convolution operation is performed between the kernels included in the (m+1)th convolution layer 56, and the output image 20 can be obtained based on this.

[0097] Fig.10 is a flowchart of an operation method of the image processing apparatus 100 according to an embodiment of the present disclosure.

[0098] Reference Fig.10 In operation S1010, the image processing apparatus 100 according to an embodiment of the present disclosure may obtain first feature information by performing a convolution operation between a first image and a first kernel included in a first convolution layer.

[0099] The image processing device 100 according to an embodiment of the present disclosure can obtain an output image by processing an input image using a CNN including n convolutional layers. Here, in each convolutional layer, a convolution operation can be performed between an image (feature information) input to the convolutional layer and a kernel included in the convolutional layer, and feature information generated as a result of the convolution operation can be output. The image processing device 100 can obtain first feature information by performing a convolution operation between a first image input to an mth convolutional layer among n convolutional layers and a first kernel included in the mth convolutional layer. Here, the mth convolutional layer may include at least one convolutional layer from the first convolutional layer to the n-1th convolutional layer, but is not limited thereto.

[0100] In operation S1020, the image processing apparatus 100 according to an embodiment of the present disclosure may obtain at least one piece of characteristic information based on the first feature information.

[0101] The image processing device 100 can obtain at least one piece of characteristic information by filtering or transforming the first feature information. Figure 3 A method of obtaining at least one piece of characteristic information by filtering the first feature information has been described, so a detailed description thereof will not be provided.

[0102] In operation S1030, the image processing apparatus 100 according to an embodiment of the present disclosure may obtain second feature information based on the first feature information and at least one piece of characteristic information.

[0103] The image processing device 100 may obtain at least one sub-feature information from at least one feature information. For example, the image processing device 100 may perform a convolution operation between at least one feature information and a sub-kernel to obtain at least one sub-feature information about at least one feature information.

[0104] The image processing apparatus 100 may obtain the second feature information by combining the first feature information, at least one piece of characteristic information, and at least one piece of sub-feature information. Figure 6 This has been described in detail, so a detailed description thereof will not be provided again.

[0105] In operation S1040, the image processing apparatus 100 may obtain third feature information by inputting the second feature information to a second convolutional layer which is a next layer of the first convolutional layer and performing a convolution operation between the second feature information and a second kernel included in the second convolutional layer.

[0106] In operation S1050, the image processing apparatus 100 may obtain an output image based on the third feature information.

[0107] Fig.11 is a block diagram of a configuration of the image processing apparatus 100 according to an embodiment of the present disclosure.

[0108] Reference Fig.11 , the image processing device 100 according to an embodiment of the present disclosure may include a processor 120 and a memory 130 .

[0109] The processor 120 according to an embodiment of the present disclosure may generally control the image processing apparatus 100. The processor 120 according to an embodiment of the present disclosure may execute at least one program stored in the memory 130.

[0110] The memory 130 according to an embodiment of the present disclosure may store various types of data, programs, or applications for driving and controlling the image processing apparatus 100. The program stored in the memory 130 may include at least one instruction. The program (e.g., at least one instruction) or application stored in the memory 130 may be executed by the processor 120.

[0111] The processor 120 according to an embodiment of the present disclosure may obtain first feature information by performing a convolution operation between the first image and a first kernel included in the first convolution layer.

[0112] The processor 120 may obtain at least one piece of characteristic information based on the first characteristic information. For example, the processor 120 may obtain at least one piece of characteristic information by filtering or transforming the first characteristic information. Figure 3 A method of obtaining at least one piece of characteristic information is described, so a detailed description thereof is not provided.

[0113] The processor 120 may obtain at least one piece of sub-feature information from at least one piece of feature information. For example, the processor 120 may perform a convolution operation between at least one piece of feature information and a sub-kernel to obtain at least one piece of sub-feature information.

[0114] Optionally, the processor 120 may obtain the second feature information based on the first feature information, at least one piece of characteristic information, and at least one piece of sub-feature information.

[0115] For example, the processor 120 may obtain the second feature information by adding or multiplying the first feature information and the at least one feature information. Optionally, the processor 120 may obtain the second feature information by adding or multiplying the first feature information and the at least one sub-feature information. Optionally, the processor 120 may obtain the second feature information by adding the at least one feature information to a value obtained by multiplying the first feature information and the at least one feature information. Optionally, the processor 120 may obtain the second feature information by adding the at least one sub-feature information to a value obtained by multiplying the first feature information and the at least one feature information. Optionally, the processor 120 may obtain the second feature information by adding the second sub-feature information to a value obtained by multiplying the first feature information and the first sub-feature information. Optionally, the processor 120 may obtain the second feature information by adding the first feature information to a value obtained by multiplying the at least one feature information and the at least one sub-feature information. Optionally, the processor 120 may obtain the second feature information by the following operation: adding a value obtained by multiplying the first feature information by the at least one feature information and a value obtained by multiplying the at least one sub-feature information by the at least one feature information. Optionally, the processor 120 may obtain the second feature information by the following operation: adding a value obtained by multiplying the first feature information and the first sub-feature information and a value obtained by multiplying the at least one feature information and the second sub-feature information. Here, the multiplication may be performed via element-by-element multiplication, and the addition may be performed via element-by-element summation.

[0116] The processor 120 may obtain the third feature information by performing a convolution operation between the second feature information and a second kernel included in a second convolution layer which is a next layer of the first convolution layer. The processor 120 may obtain an output image based on the third feature information.

[0117] Fig.12 is a block diagram of the processor 120 according to an embodiment of the present disclosure.

[0118] Reference Fig.12 , the processor 120 according to an embodiment of the present disclosure may include a first feature obtainer 1210 , a characteristic obtainer 1220 , a sub-feature obtainer 1230 , a combiner 1240 , and a third feature obtainer 1250 .

[0119] The first feature obtainer 1210 may obtain first feature information by performing a convolution operation between the first image and a first kernel included in the first convolution layer.

[0120] The characteristic obtainer 1220 may obtain at least one characteristic information based on the first characteristic information. For example, the characteristic obtainer 1220 may filter or transform the first characteristic information to extract at least one characteristic information. Figure 3 A method of obtaining at least one piece of characteristic information is described, so a detailed description thereof is not provided.

[0121] The sub-feature obtainer 1230 may obtain at least one sub-feature information from at least one feature information. For example, the sub-feature obtainer 1230 may perform a convolution operation between at least one feature information and a sub-kernel to obtain at least one sub-feature information.

[0122] The combiner 1240 may obtain the second feature information based on the first feature information, the at least one piece of characteristic information, and the at least one piece of sub-feature information.

[0123] For example, the combiner 1240 may obtain the second feature information by adding or multiplying the first feature information and the at least one feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding or multiplying the first feature information and the at least one sub-feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding the at least one feature information to a value obtained by multiplying the first feature information and the at least one feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding the at least one sub-feature information to a value obtained by multiplying the first feature information and the at least one feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding the second sub-feature information to a value obtained by multiplying the first feature information and the first sub-feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding the first feature information to a value obtained by multiplying the at least one feature information and the at least one sub-feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding a value obtained by multiplying the first feature information by the at least one feature information and a value obtained by multiplying the at least one sub-feature information by the at least one feature information. Alternatively, the combiner 1240 may obtain the second feature information by adding a value obtained by multiplying the first feature information by the first sub-feature information and a value obtained by multiplying the at least one feature information by the second sub-feature information. Here, the multiplication may be performed via element-by-element multiplication, and the addition may be performed via element-by-element summation.

[0124] The third feature obtainer 1250 may obtain the third feature information by performing a convolution operation between the second feature information and a second kernel included in a second convolutional layer which is a next layer of the first convolutional layer.

[0125] in addition, Fig.12 At least one of the first feature obtainer 1210, the feature obtainer 1220, the sub-feature obtainer 1230, the combiner 1240, or the third feature obtainer 1250 may be manufactured in the form of a hardware chip and installed on the image processing device 120. For example, at least one of the first feature obtainer 1210, the feature obtainer 1220, the sub-feature obtainer 1230, the combiner 1240, or the third feature obtainer 1250 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be installed on various image processing devices by being manufactured as a part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor) or a graphics-dedicated processor (e.g., a graphics processing unit (GPU)).

[0126] In this case, the first feature obtainer 1210, the characteristic obtainer 1220, the sub-feature obtainer 1230, the combiner 1240 and the third feature obtainer 1250 may be installed on one image processing device, or each of the first feature obtainer 1210, the characteristic obtainer 1220, the sub-feature obtainer 1230, the combiner 1240 and the third feature obtainer 1250 may be installed on a separate image processing device.

[0127] In addition, at least one of the first feature obtainer 1210, the characteristic obtainer 1220, the sub-feature obtainer 1230, the combiner 1240, or the third feature obtainer 1250 may be implemented as a software module. When at least one of the first feature obtainer 1210, the characteristic obtainer 1220, the sub-feature obtainer 1230, the combiner 1240, or the third feature obtainer 1250 is implemented as a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable medium. In addition, in this case, at least one software module may be provided by an operating system (OS) or a specific application. Alternatively, a portion of at least one software module may be provided by the OS, and the remaining portion may be provided by a specific application.

[0128] in addition, Fig.11 and Fig.12 The block diagram of the image processing device 100 and the processor 120 is only an example. Based on the specifications of the image processing device 100 actually implemented, a component may be integrated, another component may be added, or a component may be omitted. In other words, two or more components may be integrated into one component, or one component may be divided into two or more components when necessary. In addition, the functions performed by each block are used to describe the embodiments of the present disclosure, and the detailed operations or devices do not limit the scope of the present disclosure.

[0129] The operating method of the image processing device according to the embodiment of the present disclosure can be recorded on a non-transitory computer-readable recording medium by being implemented in a computer program executed by using various computers. The non-transitory computer-readable recording medium may include at least one of a program command, a data file, or a data structure. The program commands recorded in the non-transitory computer-readable recording medium may be specially designed or well known to ordinary technicians in the field of computer software. Examples of non-transitory computer-readable recording media include hardware devices specially configured to store and execute program commands (such as magnetic media (such as hard disks, floppy disks, and tapes), optical media (such as CD-ROMs and DVDs), magneto-optical media (such as optical floppy disks and ROMs, RAMs, and flash memories)). Examples of computer commands include mechanical codes prepared by a compiler, and high-level languages ​​executable by a computer using an interpreter.

[0130] In addition, the image processing apparatus and the operating method of the image processing apparatus according to the embodiments of the present disclosure may be provided by being included in a computer program product. The computer program product may be traded between a seller and a buyer as a product.

[0131] The computer program product may include a software (S / W) program or a non-transitory computer-readable storage medium storing the S / W program. For example, the computer program product may include a computer program that is distributed by a manufacturer of an electronic device or an electronic market (e.g., Google TM A product in the form of an S / W program (e.g., a downloadable application) electronically distributed through a website (e.g., Play Store or App Store). For electronic distribution, at least a portion of the S / W program may be stored in a storage medium or temporarily generated. In this case, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a forwarding server that temporarily stores the S / W program.

[0132] In a system including a server and a client device, the computer program product may include a storage medium of the server or a storage medium of the client device. Alternatively, when there is a third device (e.g., a smart phone) communicatively connected to the server or the client device, the computer program product may include a storage medium of the third device. Alternatively, the computer program product may include the S / W program itself sent from the server to the client device or the third device or from the third device to the client device.

[0133] In this case, one of the server, the client device, and the third device may execute a computer program product to execute the method according to an embodiment of the present disclosure. Alternatively, two or more of the server, the client device, and the third device may execute a computer program product to publish and execute the method according to an embodiment of the present disclosure.

[0134] For example, a server (eg, a cloud server or an AI server) may execute a computer program product stored on the server to control a client device communicatively connected to the server to perform a method according to an embodiment of the present disclosure.

[0135] The image processing device according to an embodiment of the present disclosure can perform a convolution operation by combining at least one characteristic obtained from existing feature information obtained in CNN with the existing feature information, thereby using other characteristics and features that are not obtainable from the existing feature information in the convolution operation by using CNN.

[0136] At least one piece of characteristic information according to an embodiment of the present disclosure may be intentionally obtained, and by performing a convolution operation after combining the intentionally obtained characteristic information with existing characteristic information, the existing characteristics may be maintained while processing an image in an intended direction.

[0137] The image processing device according to the embodiment of the present disclosure can improve image processing performance without significantly increasing hardware complexity.

[0138] Although the embodiments of the present disclosure have been described in detail, the scope of the present disclosure is not limited thereto, and those skilled in the art will appreciate that various changes and modifications using the basic concept of the present disclosure defined by the appended claims are also within the scope of the present disclosure.

Claims

1. An image processing device, comprising: A memory storing at least one instruction; as well as A processor configured to execute the at least one instruction stored in the memory to perform the following operations: obtaining first feature information by performing a convolution operation on the first image and a first kernel included in a first convolution layer among the plurality of convolution layers; Obtain at least one piece of characteristic information based on the first characteristic information; Obtain at least one sub-feature information based on the at least one characteristic information; Obtaining second feature information by combining the first feature information and the at least one sub-feature information; Obtaining third feature information by performing a convolution operation on the obtained second feature information and a second kernel included in a second convolutional layer among the plurality of convolutional layers, wherein the second convolutional layer is a next layer of the first convolutional layer; as well as An output image is obtained based on the third feature information.

2. The image processing apparatus according to claim 1, wherein: The processor is further configured to execute the at least one instruction to perform the following operations: obtain the at least one characteristic information from the first feature information by performing filtering or conversion on the first feature information.

3. The image processing apparatus according to claim 2, wherein: The processor is further configured to execute the at least one instruction to perform the following operations: obtain frequency characteristic information about the first feature information from the first feature information.

4. The image processing apparatus according to claim 1, wherein: The at least one piece of sub-feature information includes first sub-feature information, The processor is further configured to execute the at least one instruction to perform the following operations: The second feature information is obtained by combining the first feature information and the first sub-feature information.

5. The image processing apparatus according to claim 4, wherein: The processor is further configured to execute the at least one instruction to perform the following operations: obtain first sub-feature information from the at least one piece of feature information by performing a convolution operation on the at least one piece of feature information and the first sub-kernel.

6. The image processing apparatus according to claim 4, wherein: The processor is further configured to execute the at least one instruction to perform the following operations: obtain second feature information based on the first feature information, the at least one characteristic information and the first sub-feature information.

7. The image processing apparatus according to claim 4, wherein: The at least one sub-feature information further includes second sub-feature information, The processor is further configured to execute the at least one instruction to perform the following operations: The second feature information is obtained based on the at least one piece of characteristic information, the first feature information, the first sub-feature information, and the second sub-feature information.

8. The image processing apparatus according to claim 7, wherein: The processor is further configured to execute the at least one instruction to perform the following operations: obtain second sub-feature information from the at least one piece of feature information by performing a convolution operation on the at least one piece of feature information and the second sub-kernel.

9. The image processing apparatus according to claim 7, wherein: The processor is further configured to execute the at least one instruction to obtain the second feature information by adding a value obtained by multiplying the first feature information and the first sub-feature information and a value obtained by multiplying the at least one characteristic information and the second sub-feature information.

10. An operating method of an image processing device, the operating method comprising: obtaining first feature information by performing a convolution operation on the first image and a first kernel included in a first convolution layer among the plurality of convolution layers; Obtain at least one piece of characteristic information based on the first characteristic information; Obtain at least one sub-feature information based on the at least one characteristic information; Obtaining second feature information by combining the first feature information and the at least one sub-feature information; Obtaining third feature information by performing a convolution operation on the obtained second feature information and a second kernel included in a second convolutional layer among the plurality of convolutional layers, wherein the second convolutional layer is a next layer of the first convolutional layer; as well as An output image is obtained based on the third feature information.

11. The operating method according to claim 10, wherein: The step of obtaining the at least one piece of characteristic information includes obtaining the at least one piece of characteristic information from the first characteristic information by performing filtering or conversion on the first characteristic information.

12. The operating method according to claim 11, wherein: The step of obtaining the at least one piece of characteristic information includes obtaining frequency characteristic information about the first characteristic information from the first characteristic information.

13. The operating method according to claim 10, wherein: The at least one piece of sub-feature information includes first sub-feature information, The step of obtaining the second characteristic information includes: The second feature information is obtained by combining the first feature information and the first sub-feature information.

14. The operating method according to claim 13, wherein: The step of obtaining the first sub-feature information includes obtaining the first sub-feature information from the at least one piece of feature information by performing a convolution operation on the at least one piece of feature information and a first sub-kernel.

15. A computer program product comprising: At least one computer-readable recording medium, wherein the at least one computer-readable recording medium stores a program for executing the method according to claim 10.

Images