Image processing device and operation method thereof

Abstract

An image processing device processes an image by using one or more neural networks, and includes: a memory storing one or more instructions and a data structure for a main neural network and a sub-neural network; and a processor configured to execute the one or more instructions stored in the memory, process an input image by using the main neural network to obtain intermediate result data and a final output image, and process the intermediate result data by using the sub-neural network to output an intermediate image while the input image is being processed by using the main neural network.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on and claims the benefit of priority from Korean Patent Application No. 10-2018-0141951 filed on November 16, 2018, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein in its entirety by reference. Technical Field

[0003] The present disclosure relates to an image processing apparatus and an operating method thereof, and more particularly, to an image processing apparatus for processing an input image by using a neural network including a plurality of convolutional layers and a method for operating the image processing apparatus. Background Art

[0004] Compared to existing rule-based intelligent systems, artificial intelligence (AI) systems are computer systems that are configured to achieve human-level intelligence and spontaneously train themselves and make determinations to become more intelligent. As the recognition rate of AI systems increases and the more AI systems are used, the more accurately they understand user preferences, existing rule-based intelligent systems are gradually being replaced by deep learning AI systems.

[0005] AI technology includes machine learning (e.g., deep learning) and elemental technologies that employ machine learning. Machine learning is an algorithmic technology that self-classifies / learns the characteristics of input data and uses a machine learning algorithm (e.g., deep learning), and includes technical fields such as language understanding, visual understanding, deduction / prediction, knowledge representation, and operational control.

[0006] Various fields where AI technology is applied are as follows. Language understanding is a technology that recognizes human language / text and applies / processes human language / text, including natural language processing, machine translation, conversational systems, question answering, speech recognition / synthesis, etc. Visual understanding is a technology that recognizes and processes objects like human vision, including object recognition, object tracking, image search, human recognition, scene understanding, spatial understanding, image improvement, etc. Deduction / prediction is a technology that logically performs deduction and prediction by determining information, including knowledge / probability-based deduction, optimization prediction, preference-based planning, recommendation, etc. Knowledge representation is a technology that automatically processes human experience information into knowledge data, including knowledge establishment (data generation / classification), knowledge management (data utilization), etc. Operation control is a technology that controls the autonomous driving of vehicles and the movement of robots, including motion control (navigation, collision avoidance and driving), manipulation control (behavior control), etc. Summary of the invention

[0007] Provided are an image processing device capable of generating an intermediate output image by using a main neural network and a sub-neural network, and a method of operating the image processing device.

[0008] Additional aspects will be set forth in part in the description which follows and, in part, will be apparent from the description, or may be learned by practice of presented embodiments of the disclosure.

[0009] According to an embodiment of the present disclosure, an image processing device may include: a memory storing one or more instructions and a data structure for a main neural network and a sub-neural network; and a processor configured to execute the one or more instructions stored in the memory, process an input image by using the main neural network to obtain intermediate result data and a final output image, and process the intermediate result data by using the sub-neural network to output an intermediate image while the input image is being processed by using the main neural network to obtain the final output image.

[0010] The main neural network may include multiple first convolutional layers, and the processor may be further configured to perform a first convolution operation on the input image by using one or more kernels within the multiple first convolutional layers to extract feature information, and generate the intermediate result data based on the extracted feature information.

[0011] The sub-neural network may include one or more second convolutional layers, and the processor may be further configured to perform a second convolution operation by applying one or more kernels to the intermediate result data within the one or more second convolutional layers to extract feature information, and output the intermediate image based on the extracted feature information.

[0012] The main neural network may be a network trained to generate the final output image by applying a preset style to the input image, and the extent to which the preset style is applied to the intermediate image may be less than the extent to which the preset style is applied to the final output image.

[0013] A first time period taken to generate the final output image from the input image by using the main neural network may be greater than a second time period taken to generate the intermediate image from the input image by using the main neural network and the sub-neural network.

[0014] The processor may also be configured to predict the first time period and the second time period and determine whether to generate the intermediate image by using the sub-neural network based on the predicted first time period and the predicted second time period.

[0015] The image processing apparatus may further include a display, wherein the processor is further configured to control the display to display a time period taken to generate at least one of the intermediate image or the final output image from a current time point.

[0016] The processor may be further configured to stop processing of the input image before generating the final output image based on a user input.

[0017] The main neural network may include an input layer, an output layer, a plurality of first convolutional layers between the input layer and the output layer, and at least one intermediate output layer between two consecutive convolutional layers in the plurality of first convolutional layers. The processor may also be configured to send the intermediate result data to the sub-neural network through the at least one intermediate output layer, while processing the input image through the plurality of first convolutional layers to obtain the final output image.

[0018] According to another embodiment of the present disclosure, an image processing device may include: a communication interface configured to receive intermediate result data from a server including a main neural network while the server is processing an input image by using the main neural network; a memory storing one or more instructions and a data structure for a sub-neural network; and a processor configured to execute the one or more instructions stored in the memory to output an intermediate image by processing the intermediate result data based on the sub-neural network.

[0019] The main neural network may include an input layer, an output layer, a plurality of first convolutional layers between the input layer and the output layer, and at least one intermediate output layer between two consecutive convolutional layers in the plurality of first convolutional layers. The processor may also be configured to cause the sub-neural network to receive the intermediate result data from the at least one intermediate output layer of the main neural network while the input image is being processed by the plurality of first convolutional layers of the main neural network.

[0020] According to another embodiment of the present disclosure, a method for operating an image processing device may include: processing an input image by using a main neural network; obtaining intermediate result data based on the input image by using the main neural network, and simultaneously processing the input image to obtain a final output image; and generating an intermediate image based on the intermediate result data by using a sub-neural network.

[0021] The main neural network may include multiple first convolutional layers, and processing the input image by using the main neural network may include: extracting feature information by performing a first convolution operation on the input image and one or more kernels in the multiple first convolutional layers, and obtaining the intermediate result data may include: generating the intermediate result data based on the extracted feature information.

[0022] The sub-neural network may include one or more second convolutional layers, and generating the intermediate image may include: extracting feature information by applying the intermediate result data to one or more kernels in the one or more second convolutional layers to perform a second convolution operation; and generating the intermediate image based on the extracted feature information.

[0023] The main neural network may be a network trained to generate the final output image by applying a preset style to the input image, and the extent to which the preset style is applied to the intermediate image may be less than the extent to which the preset style is applied to the final output image.

[0024] A first time period taken to generate the final output image from the input image by using the main neural network may be greater than a second time period taken to generate the intermediate image from the input image by using the main neural network and the sub-neural network.

[0025] The method may further include: predicting the first time period and the second time period; and determining whether to generate the intermediate image by using the sub-neural network based on the predicted first time period and the predicted second time period.

[0026] The method may further include displaying on a display a period of time taken to generate at least one of the intermediate image or the final output image from a current point in time.

[0027] The method may further include, before generating the final output image, causing the main neural network to stop processing the input image based on user input.

[0028] According to another embodiment of the present disclosure, a computer program product includes a non-transitory computer-readable recording medium having a computer program recorded thereon, and when executed by a computer, the computer program can perform the method of operating an image processing apparatus. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following description taken in conjunction with the accompanying drawings, in which:

[0030] Figure 1 A process of processing an image by using a main neural network and a sub-neural network is shown according to an embodiment of the image processing apparatus;

[0031] Figure 2 is a flowchart of a method of operating an image processing apparatus according to an embodiment of the present disclosure;

[0032] Figure 3is a diagram for explaining a method in which an image processing apparatus according to an embodiment of the present disclosure processes an image by using a main neural network and a sub neural network;

[0033] Figure 4 is a diagram showing the structure of a main neural network according to an embodiment of the present disclosure;

[0034] Figure 5 shows input data, kernel, and output data of a first convolutional layer included in a main neural network according to an embodiment of the present disclosure;

[0035] Figure 6 Is used to describe Figure 5 A diagram showing a process in which the first convolutional layer performs a convolution operation;

[0036] Figure 7 is a diagram for describing a method in which an image processing apparatus according to an embodiment of the present disclosure operates;

[0037] Figure 8 An example in which an image processing apparatus according to an embodiment of the present disclosure displays an intermediate output image is shown;

[0038] Fig. 9 An example in which an image processing apparatus according to an embodiment of the present disclosure displays an intermediate output image is shown;

[0039] Fig.10 A method for an image processing device to determine whether to enter a sub-neural network according to an embodiment of the present disclosure is shown;

[0040] Fig.11 is a diagram for explaining a method in which an image processing apparatus according to an embodiment of the present disclosure generates an output image by using a main neural network and a sub neural network;

[0041] Fig.12 is a diagram showing an example in which an image processing apparatus and a server interact with each other to perform image processing according to an embodiment of the present disclosure;

[0042] Fig.13 is a block diagram for illustrating a method for training a main neural network and a sub-neural network according to an embodiment of the present disclosure;

[0043] Fig.14 is a block diagram of a structure of an image processing apparatus according to an embodiment of the present disclosure;

[0044] Fig.15 is a block diagram of a processor according to an embodiment of the present disclosure;

[0045] Fig.16is a diagram showing an example in which an image processing apparatus and a server interact with each other to train and recognize data according to an embodiment of the present disclosure; and

[0046] Fig.17 is a block diagram of a structure of an image processing apparatus according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0047] In the present disclosure, the expression “at least one of a, b, or c” indicates only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or variations thereof.

[0048] Hereinafter, terms used in the specification will be briefly described, and then the present disclosure will be described in detail.

[0049] Although currently widely used general terms are selected to describe the present disclosure in consideration of the functions of the present disclosure, these general terms may vary according to the intentions of those skilled in the art, case precedents, the emergence of new technologies, etc. Terms arbitrarily selected by the applicant of the present disclosure may also be used in specific cases. In this case, their meanings need to be given in the detailed description. Therefore, the terms must be defined based on their meanings and the content of the entire specification, rather than by simply declaring the terms.

[0050] When used in this specification, the terms "include" and / or "comprise" or "have" and / or "contain" indicate the presence of the elements, but do not exclude the presence or addition of one or more other elements. When used in this specification, the terms "unit", "machine" and "module" refer to a unit that performs at least one function or operation, and can be implemented as hardware, software, or a combination of hardware and software.

[0051] The embodiments of the present disclosure are described in detail herein with reference to the accompanying drawings so that the present disclosure can be easily performed by a person of ordinary skill in the art to which the present disclosure belongs. However, the present disclosure can be embodied in many different forms and should not be construed as being limited to the embodiments described herein. In the accompanying drawings, for simplicity of description, parts not related to the description are omitted, and the same reference numerals always represent the same elements.

[0052] Figure 1 A process in which the image processing apparatus 100 according to an embodiment of the present disclosure processes an image by using a main neural network and a sub-neural network is shown.

[0053] The image processing apparatus 100 may be any type of electronic device, such as a mobile phone, a tablet personal computer (PC), a digital camera, a camcorder, a laptop computer, a desktop computer, an e-book terminal, a digital broadcast terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation device, an MP3 player, or a wearable device.

[0054] The image processing device 100 can be implemented by using an image display device including a display. Specifically, an embodiment of the image processing device 100 can be easily implemented in an image display device (e.g., TV) having a large display. However, the embodiments of the present disclosure are not limited thereto. For example, the image processing device 100 can be fixed or mobile, or a digital broadcast receiver. The image processing device 100 can be implemented not only by using a flat display device, but also by using a curved display device with a curvature or a flexible display device with an adjustable curvature. The output resolution of the image processing device 100 can be, for example, high definition (HD), full HD, ultra HD, or a resolution clearer than ultra HD.

[0055] The term "user" used herein refers to a person who controls the function or operation of the image processing apparatus. Examples of the user may include an administrator and an installation engineer.

[0056] The image processing apparatus 100 may include a main neural network 200 trained to process images according to a preset purpose. The main neural network 200 may include, but is not limited to, a plurality of convolutional layers.

[0057] For example, the main neural network 200 may be a model trained to output a style transfer image 20 by applying a preset style 15 to an input image 10. In this case, the style may be determined based on lines, colors, materials, and atmospheres appearing on the image, but the embodiments of the present disclosure are not limited thereto.

[0058] The image processing apparatus 100 may process an input image (e.g., a first image) 10 by using the main neural network 200 to obtain a final output image (e.g., a second image) 20. The image processing apparatus 100 may obtain intermediate result data outputted when the input image 10 is being processed, and may input the intermediate result data to the sub-neural network 300. The image processing apparatus 100 may process the intermediate result data by using the sub-neural network 300 to generate an intermediate image (e.g., a third image). The sub-neural network 300 may include, but is not limited to, one or more convolutional layers.

[0059] The image processing apparatus 100 may obtain an intermediate output image 30 different from the final output image 20 by processing the intermediate result data using the sub-neural network 300 .

[0060] When processing an image by using a single network, the image processing apparatus 100 may obtain an intermediate output image 30 by using intermediate result data before obtaining a final output image 20 .

[0061] Therefore, the image processing apparatus 100 can obtain the intermediate output image 30 before generating the final output image, and from the perspective of the user of the image processing apparatus 100 , the waiting time until the final output image 20 is generated can be reduced.

[0062] Figure 2 is a flowchart of a method of operating the image processing apparatus 100 according to an embodiment of the present disclosure.

[0063] refer to Figure 2 , in operation S210, the image processing apparatus 100 may process the first image by using a main neural network.

[0064] The main neural network according to an embodiment of the present disclosure may be a model trained to process an image according to a preset purpose. The main neural network may be a convolutional neural network (CNN) including a plurality of convolutional layers. For example, the main neural network may be a model trained to output a style transfer image by applying a preset style to an input image. Alternatively, the main neural network may be a model trained to transform an input low-resolution image into a high-resolution image and output the high-resolution image. However, embodiments of the present disclosure are not limited thereto.

[0065] The image processing apparatus 100 may obtain a final output image by processing an input first image using a main neural network.

[0066] In operation S220 , the image processing apparatus 100 may obtain intermediate result data output while the first image is being processed.

[0067] The image processing apparatus 100 may obtain data that has passed through some convolutional layers included in the main neural network.

[0068] In operation S230 , the image processing apparatus 100 may generate a third image by processing the intermediate result data using the sub-neural network.

[0069] The image processing device 100 may process the intermediate result data by using a sub-neural network to obtain an intermediate output image (e.g., a third image) that is different from the final output image (e.g., the second image). In this case, when the main neural network is a model trained to apply a preset style, the degree to which the preset style is applied in the final output image may be different from the degree to which the preset style is applied in the intermediate output image. For example, the degree to which the preset style is applied in the intermediate output image may be less than the degree to which the preset style is applied in the final output image. However, embodiments of the present disclosure are not limited thereto.

[0070] Figure 3 is a diagram for explaining a method of processing an image by using a main neural network and a sub neural network, which is performed by the image processing device 100 according to an embodiment of the present disclosure.

[0071] refer to Figure 3 , the image processing device 100 may include a main neural network 200 and a sub-neural network 300.

[0072] The main neural network 200 may be a model trained to receive a first image and process the input first image according to a preset purpose to output a second image. Figure 3 As shown, the main neural network 200 may be a model trained to output a style transfer image 302 by applying a preset style to an input image 301. However, being trained for the purpose of style transfer is only an example, and the main neural network 200 may be a model trained for various purposes.

[0073] The main neural network 200 may have a structure in which data (e.g., image data) is input and processed while passing through a hidden layer, thereby outputting processed data. The main neural network 200 may be a deep neural network (DNN) including two or more hidden layers 210, 220. For example, the main neural network 200 may be a CNN including a plurality of convolutional layers.

[0074] The sub-neural network 300 may receive the intermediate result data output from the main neural network 200, and process the intermediate result data according to a preset purpose to output a third image 303. For example, the sub-neural network 300 has a structure that inputs the intermediate data and processes the intermediate data while passing through one or more convolutional layers (hidden layers), thereby outputting processed data 303 (e.g., a third image).

[0075] Now refer to Figures 4 to 6 The structure and operation of the main neural network 200 and the sub-neural network 300 are described in detail.

[0076] Figure 4 The structure of the main neural network 200 according to an embodiment of the present disclosure is shown.

[0077] refer to Figure 4, the main neural network 200 may include an input layer, an output layer, and a plurality of hidden layers, such as N convolutional layers 452 located between the input layer and the output layer. The main neural network 200 may receive input data 451 (e.g., a first image) through the input layer, and process the input data 451 through the N convolutional layers 452 to generate output data 453 (e.g., a second image), and provide the output data 453 through the output layer. The main neural network 200 may be a deep convolutional neural network including two or more convolutional layers. The main neural network 200 may also include a pooling layer, a fully connected layer, and a normalization layer as hidden layers.

[0078] The image processing apparatus 100 can extract “features” such as contours, lines, and colors from the first image by using the main neural network 200. Each of the N convolutional layers 452 included in the main neural network 200 can receive data, process the received data, and generate output data. For example, Figure 3 As shown, the image processing apparatus 100 may generate a first feature map by convolving an image (e.g., a first image) input to the first convolution layer 210 with one or more kernels or filters. The image processing apparatus 100 may also generate a second feature map by inputting the first feature map to the second convolution layer 220 and convolving the first feature map input to the second convolution layer 220 with one or more kernels or filters. Figures 4 to 6 Describe in more detail the convolution operation performed in a convolutional layer.

[0079] The front convolutional layers of the main neural network 200 can operate to extract low-level features such as edges or gradients from the input image. In the direction toward the rear convolutional layers, the rear convolutional layers can extract increasingly complex features (e.g., eyes, nose, mouth, and face).

[0080] One or more convolutional layers that receive and output feature maps within the main neural network 200 may be hidden layers (e.g., hidden convolutional layers). In the main neural network 200, processing operations other than operations that perform convolutions by applying one or more kernels to feature maps may be performed. For example, operations such as activation functions and pooling may be performed. The image processing device 100 may apply an activation function in order to change the value of a feature map extracted by performing a convolution operation to a nonlinear value, such as "having or not having a feature." Examples of activation functions may include: a linear function in which the activation is proportional to the input (i.e., the weighted sum from the neurons), a function represented by two equations f(x)=x (if x≥0) and f(x)=α(e x-1) (if x<0), an exponential linear unit (ELU) function composed of a rectified linear unit (ReLU) function defined as y=max(0,x), a Sigmoid function that takes a real value as input and outputs another value between 0 and 1, a Tanh function that compresses real-valued numbers to a range from -1 to 1, and a Softmax function that calculates the probability of an event among n different events. The image processing device 100 can perform subsampling (pooling) to reduce the size of the extracted feature map. In this case, maximum pooling, average pooling, L2 norm pooling, etc. can be used, but the embodiments of the present disclosure are not limited to this.

[0081] For example, after the first image 301 (input data) passes through N convolutional layers of the main neural network 200, the first image 301 may undergo operations such as an activation function or pooling and thus may be output as a second image 302.

[0082] refer to Figure 3 , the main neural network 200 may include N convolutional layers. In this case, the input image may be convolved with one or more kernels in each of the first to Mth convolutional layers 210, 220, ..., 255 of the main neural network 200, and thus intermediate result data may be output. The intermediate result data may be input to the sub-neural network 300.

[0083] The sub-neural network 300 may receive the intermediate result data from the main neural network 200. For example, the sub-neural network 300 includes an input layer for receiving the intermediate data, one or more convolutional layers for processing the intermediate result data, and an output layer for outputting the processed intermediate result data as output data (e.g., the third image 303).

[0084] One or more convolutional layers included in the sub-neural network 300 may be hidden layers (e.g., hidden convolutional layers). The sub-neural network 300 may perform processing operations other than operations of performing convolution by applying one or more kernels to a feature map. For example, operations such as activation functions and pooling may be performed. The image processing device 100 may apply an activation function to change the value of a feature map extracted by performing a convolution operation to a nonlinear value, such as "having or not having a feature". In this case, a tanh function or a ReLu function may be used, but embodiments of the present disclosure are not limited thereto. The image processing device 100 may perform subsampling (pooling) to reduce the size of the extracted feature map. In this case, maximum pooling, average pooling, L2 norm pooling, etc. may be used, but embodiments of the present disclosure are not limited thereto.

[0085] For example, after the intermediate result data passes through one or more convolutional layers of the sub-neural network 300 , the intermediate result data may undergo operations such as an activation function or pooling, and thus may be output as a third image 303 .

[0086] Figure 5 The input data, kernel, and output data of the first convolutional layer included in the main neural network according to an embodiment of the present disclosure are shown.

[0087] The main neural network 200 according to an embodiment of the present disclosure may include N convolutional layers 452, such as Figure 4 As shown, and reference Figure 5 , the first convolutional layer among the N convolutional layers 452 is now referred to as the first convolutional layer Conv_1.

[0088] The input data 510 (e.g., first input data) in the first convolutional layer Conv_1 may be a first image, and the size of the input data 510 may be w×h×C (i.e., width×height×channels), where w, h, and C represent width, height, and channels, respectively. For example, the first image may have a size of w×h, and the number of channels C may be, but is not limited to, 3 (e.g., R, G, and B). One or more kernels 530 (e.g., first kernels) in the first convolutional layer Conv_1 may have a size of Kw×Kh, and the number of kernels may be C×D (i.e., the number of input channels×the number of output channels), where D represents the number of output channels.

[0089] In the first convolution layer Conv_1, when a convolution operation is performed on the input data 510 and the first kernel 530, first output data 540 may be generated. Specifically, the first output data 540 may include one or more feature maps, and the size of the first output data 540 may be W×H×D, the size of one feature map may be W×H, and the number of feature maps (or the number of channels) may be D.

[0090] Figure 6 Is used to describe Figure 5 Schematic diagram of the process of performing a convolution operation on the first convolutional layer Conv_1.

[0091] For the sake of explanation, assume that Figure 6The input data 610 of the first convolutional layer Conv_1 in has a size of 5×5 and the number of channels is n. It is also assumed that the size of the kernel applied to the input data 610 is 3×3×n and the number of kernels is D. Here, n is the depth of the kernel and therefore has the same value as the number of channels of the input data 610. The number of channels of the input data 610 can be determined according to the number of kernels used by the layer before the current layer. In other words, it can be considered that one kernel includes n sub-kernels 631, 632, ..., 639, each sub-kernel has a size of 3×3, and the n sub-kernels 631, 632, ..., 639 can correspond to the n channels of the input data 610, respectively.

[0092] Figure 6 A process of extracting features of input data 610 by applying a first kernel 630 including n sub-kernels 631, 632, ..., 639 from the upper left end of the input data 610 to the lower right end thereof is shown. For example, a convolution operation may be performed by applying the first kernel 630 (3×3×n) to pixels included in an upper left end (3×3×n) region 611, 612, ..., 619 of the input data 610. In other words, the pixel values ​​included in the upper left end (3×3×n) region 611, 612, ..., 619 may be multiplied and summed by the weight values ​​included in the first kernel 630, thereby generating a single pixel value 641 mapped to the upper left end (3×3×n) region 611, 612, ..., 619.

[0093] In addition, the pixel values ​​included in the 3×3×n regions 621, 622, ..., 629 obtained by moving the upper left end (3×3×n) regions 611, 612, ..., 619 to the right by one pixel can be multiplied and summed with the weight values ​​included in the first kernel 630, thereby generating a single pixel value 642 mapped to the 3×3×n regions 621, 622, ..., 629. Using the same method as above, when the target of the convolution operation is scanned in units of pixels in the direction from left to right and from top to bottom within the input data 610, the target is multiplied and summed with the weight values ​​included in the first kernel 630, thereby generating a pixel value. Therefore, a 3×3 feature map 640 (output data) can be output. The data that is the target of the convolution operation can be scanned while moving one pixel at a time, but it can also be scanned while moving two or more pixels at a time. The number of pixels that the input data moves during scanning is called a stride, and the size of the output feature map can be determined according to the size of the stride.

[0094] refer to Figure 6, the size of the input data 610 is 5×5, but the size of the output data 640 is 3×3, which is smaller than the size of the input data 610. The main neural network includes several convolutional layers, and the size of the data is continuously reduced when passing through the several convolutional layers. When the size of the data is reduced before the features are fully extracted, the features of the input data may be lost. To prevent this, padding may be performed. Padding means increasing the size of the input data by giving a specific value (e.g., "0") to the edge of the output data so as to prevent the output data from having a reduced size. However, embodiments of the present disclosure are not limited thereto.

[0095] although Figure 6 Only the result of the convolution operation on the first kernel 630 is shown, but when the convolution operation is performed on D kernels, a 3×3×D feature map can be output. In other words, the number of channels of the output data is determined according to the number of kernels D, and thus the number of channels of the input data in the next layer can be determined.

[0096] Figure 7 is a diagram for describing a method in which an image processing apparatus according to an embodiment of the present disclosure operates.

[0097] refer to Figure 7 , the image processing apparatus 100 may generate a second image 720 according to an input first image 710 by using a main neural network 200 trained for the purpose of style transfer. The main neural network 200 may be a CNN including N convolutional layers, and the second image 720 may be an image obtained by applying a preset style 705 to the first image 710.

[0098] For example, the image processing apparatus 100 may input the first image 710 to the main neural network 200. The input first image 710 may include three channels (eg, R, G, and B channels), but embodiments of the present disclosure are not limited thereto.

[0099] The image processing device 100 may perform an operation of convolving the input image data with one or more kernels in the first convolutional layer 210. The data obtained via the convolution may be output by the first convolutional layer 210 and input to the second convolutional layer 220. According to the same method as above, in each of the N convolutional layers, the image processing device 100 may receive data from the previous convolutional layer, and perform an operation of convolving the received data with one or more kernels or filters, thereby outputting the data obtained via the convolution to the next convolutional layer. The image processing device 100 may perform an activation function operation on the data that has passed through the N convolutional layers, and may generate a final output image (e.g., a second image 720) based on the data obtained via the activation function operation. The final output image may be an image including three channels (R, G, and B channels), but embodiments of the present disclosure are not limited thereto.

[0100] The image processing apparatus 100 may obtain intermediate result data while processing the first image 710 by using the main neural network 200. The image processing apparatus 100 may obtain data that has passed through up to the nth convolution layer among the convolution layers included in the main neural network 200.

[0101] For example, Figure 7 As shown, the first image 710 can be input to the main neural network 200, and when the first image 710 passes through the first convolution layer 210 and the second convolution layer 220, the data convolved with one or more kernels or filters can be output as the first intermediate result data D1. The first image 710 can be input to the main neural network 200, and the data obtained after the first image 710 passes through the first to fifth convolution layers 210, 220, 230, 240 and 250 can be output as the second intermediate result data D2. The first image 710 can be input to the main neural network 200, and the data obtained after the first image 710 passes through the first to eighth convolution layers 210, 220, 230, 240, 250, 260, 270 and 280 can be output as the third intermediate result data D3. The first image 710 may be input to the main neural network 200, and data obtained after the first image 710 passes through the first to eleventh convolutional layers 210, 220, 230, 240, 250, 260, 270, 280, 290, 291, and 292 may be output as fourth intermediate result data D4. However, embodiments of the present disclosure are not limited thereto.

[0102] The image processing apparatus 100 may generate an intermediate output image different from the final output image 720 (ie, the second image 720 ) by processing the obtained intermediate result data using a sub-neural network.

[0103] For example, the image processing apparatus 100 may include first to fourth sub-neural networks 310, 320, 330, and 340. The main neural network 200 may include a plurality of intermediate output layers that output intermediate result data to the sub-neural networks 310, 320, 330, and 340. In particular, the main neural network 200 may include: a first intermediate output layer 10 located between the second convolution layer 220 and the third convolution layer 230; 1 , the first intermediate result data D1 is transmitted to the first sub-neural network 310; the second intermediate output layer IO is located between the fifth convolution layer 250 and the sixth convolution layer 260 2 , the second intermediate result data D2 is transmitted to the second sub-neural network 320; the third intermediate output layer IO is located between the eighth convolution layer 280 and the ninth convolution layer 290 3 , transmitting the third intermediate result data D3 to the third sub-neural network 330; and the fourth intermediate output layer IO located between the eleventh convolution layer 292 and the twelfth convolution layer 293 4 , the fourth intermediate result data D4 is transmitted to the fourth sub-neural network 340.

[0104] The image processing apparatus 100 may input the obtained first intermediate result data D1 to the first sub-neural network 310. The first sub-neural network 310 may include one or more convolution layers, and the one or more convolution layers may perform an operation of convolving the input first intermediate result data D1 with one or more kernels or filters. The image processing apparatus 100 may perform an activation function operation on the convolution data, and may generate a first intermediate output image S1 based on the data obtained via the activation function operation. The first intermediate output image S1 may be an image including three channels (e.g., R, G, and B channels), but embodiments of the present disclosure are not limited thereto.

[0105] The image processing apparatus 100 may generate second to fourth intermediate output images S2, S3, and S4 by processing the second to fourth intermediate result data D2, D3, and D4, respectively, using the second to fourth sub-neural networks 320, 330, and 340. The second to fourth intermediate output images S2, S3, and S4 are generated according to the method of generating the first intermediate output image S1 by processing the first intermediate result data D1 using the first sub-neural network 310, and thus a detailed description thereof will be omitted.

[0106] The degree to which the pattern is applied to obtain each of the first to fourth intermediate output images S1, S2, S3, and S4 may be less than that for the second image 720. For example, the degree to which the preset pattern 705 is applied to the first image 710 may increase in the direction from the first intermediate output image S1 to the fourth intermediate output image S4. The degree to which the preset pattern 705 is applied to obtain the second image 720 (i.e., the final output image) may be greater than the degree to which the preset pattern 705 is applied to obtain the fourth intermediate output image S4.

[0107] Figure 8 and Fig. 9 is a view showing an example in which the image processing apparatus according to the embodiment of the present disclosure displays an intermediate output image.

[0108] refer to Figure 8 , when the image processing apparatus 100 generates a first intermediate output image S1 according to the first image 710 by using the main neural network 200 and the first sub-neural network 310, a first time period T1 may be spent. The image processing apparatus 100 may display the generated first intermediate output image S1 on the display. When the image processing apparatus 100 generates a second intermediate output image S2 according to the first image 710 by using the main neural network 200 and the second sub-neural network 320, a second time period T1+T2 may be spent, and when the image processing apparatus 100 generates a third intermediate output image S3 according to the first image 710 by using the main neural network 200 and the third sub-neural network 330, a third time period T1+T2+T3 may be spent.

[0109] When the image processing apparatus 100 generates the final output image 720 (ie, the second image) from the first image 710 by using the main neural network 200, a fourth time period T4 may be spent. The fourth time period T4 may be longer than the third time period T1+T2+T3.

[0110] Before the image processing apparatus 100 generates the final output image 720 by processing the first image 710 using the main neural network 200, the image processing apparatus 100 may generate first to third intermediate output images S1, S2, and S3 and display them on a display.

[0111] The image processing apparatus 100 may display a plurality of pieces of time information 810 and 820 together with the output image, each piece of time information indicating a time period to be spent until the next output image is generated. Figure 8As shown, while displaying the first intermediate output image S1, the image processing apparatus 100 may also display time information 810, which indicates the time period T2 from the current time point to the time point at which the second intermediate output image S2 will be generated. While displaying the second intermediate output image S2, the image processing apparatus 100 may also display time information 820, which indicates the time period T3 until the third intermediate output image S3 is generated.

[0112] The image processing apparatus 100 may display not only time information indicating a time period that will be taken until the next intermediate output image is generated, but also time information indicating a time period that will be taken until the final output image is generated.

[0113] Therefore, the user can check the intermediate output image displayed on the display before the final output image is generated, and can check the time period to be taken until the next output image is generated, thereby determining whether the image processing is continuously performed.

[0114] The image processing apparatus 100 may interrupt the image processing of the first image 710 in the main neural network 200 based on the user input. For example, when the third intermediate output image S3 is displayed and an image processing interruption request is received at this time, the image processing apparatus 100 may interrupt the image processing of the first image 710 in the main neural network 200. Therefore, the image processing apparatus 100 may not generate the next intermediate output image and the final output image 720.

[0115] refer to Fig. 9 , the image processing device 100 may perform additional image processing on the intermediate output image based on the user input. For example, the image processing device 100 may generate the final image 920 by performing image processing (sharpening) that adjusts the color sense, brightness, clarity, etc. of the third intermediate output image S3. However, the embodiments of the present disclosure are not limited thereto.

[0116] Fig.10 A method of determining whether to enter a sub-neural network, which is performed by the image processing apparatus 100, is shown.

[0117] refer to Fig.10, the image processing apparatus 100 may determine the points in the main neural network 200 where each intermediate data is obtained as a sub-network point. For example, the image processing apparatus 100 may determine the output point of the second convolutional layer (the input point of the third convolutional layer) where the first intermediate result data D1 is obtained as the first sub-network point 1010, the output point of the fifth convolutional layer (the input point of the sixth convolutional layer) where the second intermediate result data D2 is obtained as the second sub-network point 1020, the output point of the eighth convolutional layer (the input point of the ninth convolutional layer) where the third intermediate result data D3 is obtained as the third sub-network point 1030, and the output point of the eleventh convolutional layer (the input point of the twelfth convolutional layer) where the fourth intermediate result data D4 is obtained as the fourth sub-network point 1040. However, the embodiments of the present disclosure are not limited thereto.

[0118] The image processing apparatus 100 may determine whether to perform image processing by using the sub-neural network at the first to fourth sub-network points 1010, 1020, 1030, and 1040. For example, when the convolution operation is completed in the first convolution layer and the second convolution layer and thus the first intermediate result data D1 is output (at the first sub-network point 1010), the image processing apparatus 100 may input the first intermediate result data D1 to the first sub-neural network 310 and perform image processing, thereby determining whether to generate the first intermediate output image S1.

[0119] The image processing apparatus 100 may calculate and estimate a time period (e.g., a first time period) to be spent processing the first intermediate result data D1 into the first intermediate output image S1 by using the first sub-neural network 310 and a time period (e.g., a second time period) to be spent processing the first intermediate result data D1 into the final output image 720 by using the main neural network 200. When the difference between the first time period and the second time period is equal to or greater than a preset value, the image processing apparatus 100 may enter the first sub-neural network 310. For example, the image processing apparatus 100 may generate the first intermediate output image S1 using the first intermediate result data D1 and the first sub-neural network 310. On the other hand, when the difference between the first time period and the second time period is less than a preset value, the image processing apparatus 100 may not enter the first sub-neural network 310. For example, without generating the first intermediate output image S1, the image processing apparatus 100 may generate the final output image 720 using the first intermediate result data D1 and the main neural network 200.

[0120] Likewise, the image processing apparatus 100 may calculate a time period (e.g., a third time period) required to process the second intermediate result data D2 into the second intermediate output image S2 by using the second sub-neural network 320 and a time period (e.g., a fourth time period) required to process the second intermediate result data D2 into the final output image 720 by using the main neural network 200. The image processing apparatus 100 may determine whether to enter the second sub-neural network 320 by comparing the third time period and the fourth time period with each other at the second sub-network point 1020.

[0121] The image processing device 100 can also determine whether to enter the third sub-neural network 330 by comparing at the third sub-network point 1030 the time period (e.g., the fifth time period) required to process the third intermediate result data D3 into the third intermediate output image S3 by using the third sub-neural network 330 with the time period (e.g., the sixth time period) required to process the third intermediate result data D3 into the final output image 720 by using the main neural network 200.

[0122] The image processing device 100 can also determine whether to enter the fourth sub-neural network 340 by comparing at the fourth sub-network point 1040 the time period (e.g., the seventh time period) required to process the fourth intermediate result data D4 into the fourth intermediate output image S4 by using the fourth sub-neural network 340 with the time period (e.g., the eighth time period) required to process the fourth intermediate result data D4 into the final output image 720 by using the main neural network 200.

[0123] Fig.11 is a diagram for explaining a method of generating an output image by using a main neural network and a sub neural network, which is performed by an image processing device according to an embodiment of the present disclosure.

[0124] Fig.11 The main neural network 1100 may correspond to Figure 7 The main neural network 200, Fig.11 The first sub-neural network 1110 may correspond to Figure 7 The first sub-neural network 310 of Fig.11 The second sub-neural network 1120 may correspond to Figure 7 Therefore, a detailed description of the structures and operations of the main neural network 1100 and the first and second sub-neural networks 1110 and 1120 will be omitted here.

[0125] refer to Fig.11 , the image processing apparatus 100 may generate a second image 720 (ie, a final output image) according to the first image 710 by using the main neural network 1100 .

[0126] The image processing apparatus 100 may obtain first intermediate result data D1 by processing the first image 710 using the main neural network 1100, and may generate a first intermediate output image S1 by using the first intermediate result data D1 and the first sub neural network 1110. The image processing apparatus 100 may also obtain second intermediate result data D2 by processing the first image 710 using the main neural network 1100, and may generate a second intermediate output image S2 by using the second intermediate result data D2 and the second sub neural network 1120.

[0127] The first image 710 and the second image 720 (ie, final output images) may be high-resolution images, and the first intermediate output image S1 and the second intermediate output image S2 may be low-resolution images.

[0128] Fig.12 : is a diagram illustrating an example of interactive operation between an image processing apparatus and a server according to an embodiment of the present disclosure.

[0129] refer to Fig.12 , the server 2000 may include a main neural network 2030, and the main neural network 2030 may be a model trained to process an image according to a preset purpose. For example, the main neural network 2030 may be a model trained for the purpose of style transfer.

[0130] The server 2000 may generate a second image 720 (ie, a final image, a style transfer image) by applying a preset style to the input first image 710 using the main neural network 2030 .

[0131] because Fig.12 The main neural network 2030 may correspond to Figure 3 The server 2000 may transmit the second image 720 to the image processing apparatus 100.

[0132] The server 2000 may obtain the intermediate result data while the first image 710 is being processed in the main neural network 2030. For example, the intermediate result data may include a feature map (e.g., feature information) extracted from a specific layer among a plurality of convolutional layers included in the main neural network 2030.

[0133] The server 2000 may transmit the intermediate result data to the image processing apparatus 100 .

[0134] The image processing apparatus 100 may include a sub-neural network 2050. The sub-neural network 2050 may include one or more convolutional layers, and a convolution operation may be performed in the one or more convolutional layers. For example, the image processing apparatus 100 may input the intermediate result data received from the server 2000 into the sub-neural network 2050, and the intermediate result data may be convolved with one or more kernels while passing through one or more convolutional layers. The image processing apparatus 100 may perform an activation function operation on the convolutional data, and may generate a third image 1250 (e.g., an intermediate output image) based on the data obtained via the activation function operation.

[0135] The image processing apparatus 100 may display the second image 720 received from the server 2000 and the generated third image 1250 on a display. The degree to which the preset style is applied to the third image 1250 (e.g., an intermediate output image) may be less than the degree to which the preset style is applied to the second image 720 (e.g., a final output image).

[0136] like Fig.12 As shown, the operation of the main neural network 2030 is performed in the server 2000, and the operation of the sub-neural network 2050 is performed in the image processing device 100, so the memory bandwidth of the image processing device 100 and the time period required to obtain the output image can be reduced.

[0137] Fig.13 Detailed description is a block diagram for illustrating a method for training a main neural network and a sub-neural network according to an embodiment of the present disclosure.

[0138] refer to Fig.13 , the image processing apparatus 100 may include an image feature extractor 1310, a style analyzer 1320, a content analyzer 1330, and a network refiner 1340. The image feature extractor 1310 may receive an image Y, a style target image Ys, and a content target image Yc from the main neural network 200. The style target image Ys may be an image including a style to be transferred by using the main neural network 200. The content target image Yc may be the same image as the image X input to the main neural network 200, as an original image before the style is transferred.

[0139] The image feature extractor 1310 may include a CNN, and may extract feature maps (feature information) of the received style target image Ys, the received image Y, and the received content target image Yc by using the CNN. The CNN may include a plurality of convolutional layers, and convolution operations may be performed in the plurality of convolutional layers. In addition to the convolution operations in the plurality of convolutional layers, the image feature extractor 1310 may also perform operations such as activation functions and pooling, and may extract feature maps in each convolutional layer. For example, the image feature extractor 1310 may extract a feature map of the style target image Ys, a feature map of the content target image Yc, and a feature map of the image Y in each convolutional layer included in the image feature extractor 1310.

[0140] The image feature extractor 1310 may extract low-level features of the input image, such as edges or gradients, in the front convolution layer, and may extract gradually more complex feature maps in the rear convolution layer compared to the front convolution layer. However, embodiments of the present disclosure are not limited thereto.

[0141] The style analyzer 1320 may analyze the difference between the style of the style target image Ys and the style of the image Y. For example, the style analyzer 1320 may analyze the difference between the feature map of the style target image Ys and the feature map of the image Y.

[0142] The content analyzer 1330 may analyze the difference between the content of the content target image Yc and the content of the image Y. For example, the content analyzer 1330 may analyze the difference between the feature map of the content target image Yc and the feature map of the image Y.

[0143] The network refiner 1340 may adjust the weights of the kernels of the main neural network 200 so that the difference between the feature map of the style target image and the feature map of the output image is reduced. The network refiner 1340 may also adjust the weights of the kernels of the main neural network 200 so that the difference between the feature map of the content target image and the feature map of the output image is reduced. However, embodiments of the present disclosure are not limited thereto.

[0144] although Fig.13 Only the method of training the main neural network 200 according to the embodiment of the present disclosure is shown and described, but the sub-neural network according to the embodiment of the present disclosure may be trained using the same method as described above.

[0145] Specifically, the image processing apparatus according to an embodiment of the present disclosure may include a neural network that outputs different results although partially processing an image in a similar manner, as a main neural network and a sub-neural network. For example, one of the first neural network and the second neural network may be included as the main neural network, and only a portion of the other neural network that has a different process from the main neural network may be included as the sub-neural network. However, the embodiments of the present disclosure are not limited thereto. Therefore, the image processing apparatus can improve the efficiency of neural network training by not repeating training for the same process.

[0146] Fig.14 is a block diagram of a structure of an image processing apparatus according to an embodiment of the present disclosure.

[0147] refer to Fig.14 , the image processing apparatus 100 according to an embodiment of the present disclosure may include a processor 120 and a memory 130 .

[0148] The processor 120 may execute one or more programs stored in the memory 130. The processor 120 may include a single core, a dual core, a triple core, a quad core, or a multi-core. The processor 120 may include a plurality of processors. For example, the processor 120 may be implemented by using a main processor and a sub-processor operating in a sleep mode.

[0149] The memory 130 may store various data, programs, or applications for driving and controlling the image processing apparatus 100 .

[0150] The program stored in the memory 130 may include one or more instructions. The program (one or more instructions) or application stored in the memory 130 may be executed by the processor 120.

[0151] The processor 120 may execute one or more instructions stored in the memory 130 to process the first image by using a main neural network. The main neural network may be a model trained to process an image according to a preset condition. The main neural network may be a CNN including a plurality of convolutional layers. For example, the main neural network may be a model trained to output a style transfer image by applying a preset style to an input image. Alternatively, the main neural network may be a model trained to transform an input low-resolution image into a high-resolution image and output the high-resolution image. However, embodiments of the present disclosure are not limited thereto.

[0152] The processor 120 may process the first image by using the main neural network to obtain a final output image (eg, a second image).

[0153] The processor 120 may obtain intermediate result data while processing the first image, and process the obtained intermediate result data by using the sub-neural network. The processor 120 may obtain an intermediate output image (e.g., a third image) different from the final output image (e.g., the second image) by processing the intermediate result data using the sub-neural network.

[0154] In particular, when the main neural network is a model trained to apply a preset style, the degree to which the preset style has been applied to obtain the final output image may be different from the degree to which the preset style has been applied to obtain the intermediate output image. For example, the degree to which the preset style has been applied to obtain the intermediate output image may be less than the degree to which the preset style has been applied to obtain the final output image. However, embodiments of the present disclosure are not limited thereto.

[0155] The processor 120 may predict a time period (e.g., a first time period) taken to generate an intermediate output image from a first image (e.g., an input image) and a time period (e.g., a second time period) taken to generate a final output image from the first image (e.g., the input image). The processor 120 may determine whether to generate the intermediate output image by using a sub-neural network by comparing a difference between the predicted first time period and the second time period with a preset time.

[0156] The processor 120 may control the display to display a time period taken to generate an intermediate output image from the current moment or a time period taken to generate a final output image from the current moment.

[0157] Before generating the final output image, the processor 120 may terminate the processing of the first image by using the main neural network based on the user input. The processor 120 may allow the user to stop the processing of the first image through the user interface while providing the intermediate output image to the user. For example, when the intermediate output image is displayed and an image processing interrupt request is received at this time, the processor 120 may interrupt the processing of the first image in the main neural network. Therefore, the processor 120 may not generate the next intermediate output image and the final output image.

[0158] Fig.15 is a block diagram of the processor 120 according to an embodiment of the present disclosure.

[0159] refer to Fig.15 , the processor 120 may include a data learner 1410 and a data processor 1420 .

[0160] According to an embodiment of the present disclosure, the data learner 1410 can learn the standard of image processing according to a preset purpose in order to train the main neural network and the sub-neural network. For example, the data learner 1410 can learn the standard of applying a preset style to the input image using which information (e.g., feature information) of the input image.

[0161] The data learner 1410 can learn the standard on how to apply the preset style by using the feature information of the image. The data learner 1410 can learn the standard for applying the preset style to the input image by obtaining data (e.g., image) for learning and applying the obtained data to the data processing model (main neural network and sub neural network).

[0162] The data processing model (e.g., main neural network and sub-neural network) may be established in consideration of, for example, the application field of the recognition model, the purpose of learning, or the computer performance of the device. The data processing model may be, for example, a neural network-based model. For example, a model such as a deep neural network (DNN), a recurrent neural network (RNN), or a bidirectional recurrent DNN (BRDNN) may be used as a data processing model, but embodiments of the present disclosure are not limited thereto.

[0163] The data learner 1410 may train the data processing model by using a learning algorithm including, for example, error back propagation or gradient descent.

[0164] For example, the data learner 1410 may train the data processing model through supervised learning by using the learning data as an input value. For example, the data learner 1410 may train the data processing model through unsupervised learning to find a standard for data processing by self-training on the data type required for data processing without supervision. For example, the data learner 1410 may train the data processing model through reinforcement learning using feedback on whether the result of the data processing according to the training is correct.

[0165] When training the data processing model, the data learner 1410 may store the trained data processing model. In this case, the data learner 1410 may store the trained data processing model in the memory of the image processing device. Alternatively, the data learner 1410 may store the trained data processing model in the memory of a server connected to the image processing device via a wired or wireless network.

[0166] Specifically, the memory storing the trained data processing model may also store, for example, commands or data related to at least one other component of the image processing device. The memory may also store software and / or programs. For example, the program may include a kernel, middleware, an application programming interface (API), and / or an application program (or application).

[0167] The data processor 1420 may input an image to a data processing model including a trained main neural network, and the data processing model may output an image processed according to a preset purpose as a result value. For example, the data processing model may output an image obtained by applying a preset style to the input image. The output result value (image) may be used to update the data processing model including the main neural network and the sub-neural network.

[0168] At least one of the data learner 1410 or the data processor 1420 may be manufactured in the form of at least one hardware chip and may be installed on the image processing device. For example, at least one of the data learner 1410 or the data processor 1420 may be manufactured in the form of a dedicated hardware chip for artificial intelligence (AI), or may be manufactured as part of an existing general-purpose processor (e.g., a central processing unit (CPU) or an application processor (AP)) or a processor dedicated to graphics (e.g., a graphics processing unit (GPU)), and may be installed on any of the various image processing devices described above.

[0169] The data learner 1410 and the data processor 1420 may both be installed on a single image processing device, or may be installed on separate image processing devices, respectively. For example, one of the data learner 1410 and the data processor 1420 may be included in the image processing device, and the other may be included in the server. The data learner 1410 and the data processor 1420 may be connected to each other by wire or wirelessly, so that the model information established by the data learner 1410 may be provided to the data processor 1420, and the data input to the data processor 1420 may be provided to the data learner 1410 as additional learning data.

[0170] At least one of the data learner 1410 or the data processor 1420 may be implemented as a software module. When at least one of the data learner 1410 or the data processor 1420 is implemented using a software module (or a program module including instructions), the software module may be stored in a non-transitory computer-readable recording medium. In this case, at least one software module may be provided by an operating system (OS) or by a specific application. Alternatively, some of the at least one software module may be provided by the OS, and others may be provided by a specific application.

[0171] Fig.16 is a diagram illustrating an example in which an image processing apparatus and a server interact with each other to learn and recognize data according to an embodiment of the present disclosure.

[0172] refer to Fig.16 , the server 2000 can train the main neural network and the sub-neural network by learning the standards for image processing according to the preset purpose.

[0173] In this case, Server 2000 can execute Fig.15 The server 2000 can learn the standard for processing the input image according to the preset purpose using which learning data. The server 2000 can learn the standard for processing the image according to the preset purpose by using the data.

[0174] Server 2000 may train at least one of the main neural network or the sub-neural networks by using the learning data.

[0175] The image processing apparatus 100 may send data to the server 2000, and may request the server 2000 to process the data by applying the data to the data processing model (main neural network and sub neural network). For example, the server 2000 may process the image according to a preset purpose by using the data processing model (main neural network and sub neural network). For example, the server 2000 may output a style transfer image by applying a preset style to the input image.

[0176] Alternatively, the image processing apparatus 100 may receive a data processing model generated by the server 2000 from the server 2000, and may process the data by using the received data processing model. For example, the image processing apparatus 100 may process the image according to a preset purpose by using the received data processing model (main neural network and sub-neural network). For example, the image processing apparatus 100 may output a style transfer image by applying a preset style to an input image.

[0177] Fig.17 is a block diagram showing a structure of an image processing apparatus 1700 according to another embodiment of the present disclosure. Fig.17 The image processing device 1700 may be Fig.14 An embodiment of an image processing apparatus 100 .

[0178] refer to Fig.17 , the image processing device 1700 may include a tuner 1740 , a controller 1710 , a display 1720 , a communication interface 1750 , a sensor 1730 , an input / output (I / O) interface 1770 , a video processor 1780 , an audio processor 1715 , a storage device 1790 , and a power supply 1760 .

[0179] The tuner 1740 can tune and select only the frequency of a channel that the image processing apparatus 1700 wants to receive among many radio wave components obtained through amplification, mixing, resonance, etc. of wired or wireless broadcast signals. The broadcast signal includes audio, video, and additional information (e.g., Electronic Program Guide (EPG)).

[0180] The tuner 1740 may receive a broadcast signal in a frequency band corresponding to a channel number according to a user input (eg, a control signal received from a control device such as a channel number input, a channel up / down input, and a channel input on an EPG screen image).

[0181] The tuner 1740 may receive broadcast signals from various sources such as terrestrial broadcasting, cable broadcasting, satellite broadcasting, and Internet broadcasting. The tuner 1740 may also receive broadcast signals from sources such as analog broadcasting or digital broadcasting.

[0182] The communication interface 1750 may transmit data or signals to an external device or server, or receive data or signals from an external device or server under the control of the controller 1710. The controller 1710 may transmit / receive content to / from an external device connected via the communication interface 1750, download an application from the external device, or perform web browsing. The communication interface 1750 may transmit or receive data or signals according to at least one method of a wireless local area network (LAN) 1751 (e.g., Wi-Fi), a Bluetooth network 1752, or a wired Ethernet network 1753, corresponding to the performance and structure of the image processing device 1700.

[0183] The video processor 1780 processes the video data received by the image processing apparatus 1700. The video processor 1780 may perform various image processing such as decoding, scaling, noise filtering, frame rate conversion, and resolution conversion on the received video data.

[0184] The sensor 1730 senses a user's voice, a user's image, or an interaction with a user, and may include a microphone 1731 , a camera 1732 , and a light receiver 1733 .

[0185] The microphone 1731 receives a voice uttered by the user. The microphone 1731 may convert the received voice into an electric signal and output the electric signal to the controller 1710. The user voice may include, for example, a voice corresponding to a menu or function of the image processing apparatus 1700.

[0186] The camera 1732 may receive images (e.g., continuous frames) corresponding to user actions including gestures within the recognition range of the camera 1732. The controller 1710 may select a menu displayed on the image processing device 1700 by using the received recognition result of the action, or perform control corresponding to the action recognition result.

[0187] The optical receiver 1733 receives an optical signal (including a control signal) from an external control device through a light window of a frame of the display 1720 or the like. The optical receiver 1733 may receive an optical signal corresponding to a user input (e.g., touch, press, touch gesture, voice, or action) from the control device. The control signal may be extracted from the received optical signal under the control of the controller 1710.

[0188] Under the control of the controller 1710, the I / O interface 1770 receives video (e.g., moving images), audio (e.g., voice or music), and additional information (e.g., EPG) from the outside of the image processing apparatus 1700. The I / O interface 1770 may include a high-definition multimedia interface (HDMI) port 1771, a component jack 1772, a PC port 1773, or a USB port 1774. The I / O interface 1770 may include a combination of the HDMI port 1771, the component jack 1772, the PC port 1773, and the USB port 1774.

[0189] The controller 1710 controls the overall operation of the image processing device 1700 and signal transmission between internal components of the image processing device 1700, and processes data. The controller 1710 may execute the OS and various applications stored in the storage device 1790 when there is a user input or a stored preset condition is satisfied.

[0190] The controller 1710 may include a random access memory (RAM) 1781, a read-only memory (ROM) 1782, and a processor 1783, wherein the RAM 1781 stores signals or data input by an external source of the image processing device 1700 or is used as a memory area for various operations performed by the image processing device 1700, and the ROM 1782 stores a control program for controlling the image processing device 1700.

[0191] The graphics processor 1784 generates a screen image including various objects (e.g., icons, images, and text) by using an arithmetic unit and a rendering unit. The arithmetic unit calculates attribute values ​​to be displayed for each object according to the layout of the screen image, such as coordinate values, shapes, sizes, colors, etc., based on the user interaction sensed by the sensor 1730. The rendering unit generates screen images of various layouts including objects based on the attribute values ​​calculated by the arithmetic unit. The screen image generated by the rendering unit is displayed on the display area of ​​the display 1720.

[0192] The first through nth interfaces 1785-1 through 1785-n are connected to the above-mentioned components of the image processing apparatus 1700. One of the first through nth interfaces 1785-1 through 1785-n may be a network interface connected to an external apparatus via a network.

[0193] The RAM 1781 , the ROM 1782 , the processor 1783 , the graphic processor 1784 , and the first to nth interfaces 1785 - 1 to 1785 - n may be connected to one another via an internal bus 1786 .

[0194] The term “controller of the image processing apparatus” used in the present embodiment of the present disclosure includes a processor 1783 , a ROM 1782 , and a RAM 1781 .

[0195] The display 1720 may generate a driving signal by converting an image signal, a data signal, an on-screen display (OSD) signal, and a control signal processed by the controller 1710. The display 1720 may be a plasma display panel (PDP), a liquid crystal display (LCD), an organic light emitting device (OLED), a flexible display, or a three-dimensional (3D) display. The display 1720 may be configured as a touch screen and thus may be used as an input device as well as an output device.

[0196] The audio processor 1715 processes audio data. The audio processor 1715 may perform various processes on the audio data, such as decoding, amplification, or noise filtering. The audio processor 1715 may include a plurality of audio processing modules to process audio corresponding to a plurality of pieces of content.

[0197] Under the control of the controller 1710, the audio output interface 1725 outputs audio included in a broadcast signal received via the tuner 1740. The audio output interface 1725 may also output audio (e.g., voice or sound) input via the communication interface 1750 or the I / O interface 1770. Under the control of the controller 1710, the audio output interface 1725 may also output audio stored in the storage device 1790. The audio output interface 1725 may include at least one selected from a speaker 1726, a headphone output port 1727, and a Sony / Philips Digital Interface (S / PDIF) output port 1728. The audio output interface 1725 may include a combination of the speaker 1726, the headphone output port 1727, and the S / PDIF output port 1728.

[0198] Under the control of controller 1710, power supply 1760 supplies power input from an external power source to internal components of image processing device 1700. Under the control of controller 1710, power supply 1760 may also supply power output by one or more batteries located in image processing device 1700 to internal components of image processing device 1700.

[0199] The storage device 1790 may store various data, programs or applications for driving and controlling the image processing device 1700 under the control of the controller 1710. The storage device 1790 may include a broadcast receiving module, a channel control module, a volume control module, a communication control module, a voice recognition module, a motion recognition module, a light receiving module, a display control module, an audio control module, an external input control module, a power control module, a power control module of an external device connected in a wireless manner (e.g., Bluetooth), a voice database (DB) or a motion DB. These modules and DB of the storage device 1790 may be implemented as software so as to perform a broadcast receiving control function, a channel control function, a volume control function, a communication control function, a voice recognition function, a motion recognition function, a light receiving control function, a display control function, an audio control function, an external input control function, a power control function or a power control function of an external device connected in a wireless manner (e.g., Bluetooth) of the image processing device 1700. The controller 1710 may perform these functions by using the software stored in the storage device 1790.

[0200] Fig.14 and Fig.17 The block diagrams of the image processing apparatuses 100 and 1700 shown in FIG. 1 are merely exemplary embodiments of the present disclosure. When actually implemented, the image processing apparatuses 100 and 1700 may be combined or omitted according to the specifications. Fig.14 and Fig.17 Components shown in or can be Fig.14 and Fig.17 In other words, two or more components may be combined into a single component, or a single component may be divided into two or more components. The functions performed in each block are merely examples for illustrating embodiments of the present disclosure, and the detailed operations or devices of each block do not limit the scope of embodiments of the present disclosure.

[0201] The method of operating an image processing device according to an embodiment of the present disclosure can be implemented as a program command that can be executed by various computer devices, and can be recorded on a computer-readable recording medium. The computer-readable recording medium may include program commands, data files, data structures, etc., alone or in combination. The program commands to be recorded on the computer-readable recording medium may be specially designed and configured for the embodiments of the present disclosure, or may be well known and available to ordinary technicians in the field of computer software. Examples of computer-readable recording media include: magnetic media such as hard disks, floppy disks, and tapes; optical media such as compact disk read-only memories (CD-ROMs) or digital versatile disks (DVDs); magneto-optical media such as optical magnetic floppy disks; and hardware devices such as ROMs, random access memories (RAMs), or flash memories that are specially configured to store and execute program commands. Examples of program commands are high-level language codes that can be executed by a computer by using an interpreter, etc., and machine language codes generated by a compiler.

[0202] Furthermore, the image processing apparatus or the method of operating the image processing apparatus according to the disclosed embodiments may be provided in a computer program product. The computer program product may be traded as a product between a seller and a buyer.

[0203] A computer program product may include a software program and a computer-readable storage medium on which the software program is stored. For example, a computer program product may include a product in the form of a software program (e.g., a downloadable application) distributed electronically by a manufacturer of an electronic device or an electronic market (e.g., Google PlayStore, AppStore). For electronic distribution, at least a portion of the software program may be stored on a storage medium, or may be temporarily created. In this case, the storage medium may be a storage medium of a manufacturer's server, a server of an electronic market, or a relay server for temporarily storing SW programs.

[0204] In a system including a server and a client device, the computer program product may include the storage medium of the server or the storage medium of the client device. Alternatively, when there is a third device (e.g., a smart phone) communicating with the server or the client device, the computer program product may include the storage medium of the third device. Alternatively, the computer program product may include the S / W program itself sent from the server to the client device or the third device or from the third device to the client device.

[0205] In this case, one of the server, the client device, and the third device may execute the computer program product to perform the method according to the disclosed embodiment. Alternatively, two of the server, the client device, and the third device may execute the computer program product to distribute and perform the method according to the disclosed embodiment.

[0206] For example, a server (e.g., a cloud server or an artificial intelligence server) may execute a computer program product stored on the server to control a client device communicating with the server to perform a method according to the disclosed embodiments.

[0207] The image processing device according to an embodiment of the present disclosure can generate an intermediate output image by using a sub-neural network before generating a final output image in a main neural network, so the waiting time can be reduced from the user's perspective.

[0208] The image processing device can generate an intermediate output image, and the user can determine whether to generate a final output image in the main neural network based on the intermediate output image.

[0209] The image processing apparatus may include neural networks that output different results although partially processing an image similarly to each other, as a main neural network and a sub-neural network, thereby improving the efficiency of neural network training.

[0210] Although not limited thereto, the example embodiments may be implemented as computer-readable codes on a computer-readable recording medium. A computer-readable recording medium is any data storage device that can store data that can be read by a computer system afterwards. Examples of computer-readable recording media include read-only memory (ROM), random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage devices. The computer-readable recording medium may also be distributed on a networked computer system so that the computer-readable code is stored and executed in a distributed manner. In addition, the example embodiments may be written as a computer program that is transmitted via a computer-readable transmission medium (e.g., a carrier wave) and is received and executed in a general-purpose or special-purpose digital computer that executes the program. In addition, it is understood that in the example embodiments, one or more units of the above-mentioned devices and apparatus may include circuits, processors, microprocessors, etc., and may execute a computer program stored in a computer-readable medium.

[0211] The above exemplary embodiments are merely exemplary and should not be construed as limiting. The present teaching can be easily applied to other types of devices. In addition, the description of the exemplary embodiments is intended to be illustrative rather than to limit the scope of the claims, and a variety of substitutions, modifications and variations will be apparent to those skilled in the art.

Claims

1. An image processing device, comprising: A memory storing one or more instructions and a data structure for a main neural network and a sub-neural network, wherein the main neural network includes an input layer, an output layer, and a plurality of hidden layers between the input layer and the output layer; as well as a processor configured to execute the one or more instructions stored in the memory to: Processing the input image by using the main neural network to obtain intermediate result data, and processing the intermediate result data output from one of the plurality of hidden layers of the main neural network by the sub-neural network to output an intermediate image while the input image is being processed by using the main neural network to obtain a final output image, Wherein, the sub-neural network includes one or more convolutional layers, and Wherein, the processor is also configured to: predict a first time period taken to generate the final output image based on the input image and a second time period taken to generate the intermediate image based on the input image, and determine whether to generate the intermediate image by using the sub-neural network by comparing the difference between the predicted first time period and the second time period with a preset time.

2. The image processing device according to claim 1, wherein: The main neural network includes a plurality of first convolutional layers as the plurality of hidden layers, and The processor is also configured to perform a first convolution operation on the input image by using one or more kernels within the plurality of first convolutional layers to extract feature information, and output the intermediate result data based on the extracted feature information.

3. The image processing device according to claim 1, wherein: The processor is also configured to perform a second convolution operation by applying one or more kernels to the intermediate result data within the one or more convolution layers to extract feature information, and generate the intermediate image based on the extracted feature information.

4. The image processing device according to claim 1, wherein: The processor is further configured to generate the final output image.

5. The image processing device according to claim 1, wherein: The main neural network is a network trained to generate the final output image by applying a preset pattern to the input image, and the degree to which the preset pattern is applied to the intermediate image is less than the degree to which the preset pattern is applied to the final output image.

6. The image processing apparatus according to claim 1, further comprising a display, wherein: The processor is further configured to control the display to display a time period taken to generate at least one of the intermediate image or the final output image from a current time point.

7. The image processing device according to claim 1, wherein: The processor is further configured to stop processing the input image before generating the final output image based on user input.

8. A method of operating an image processing device, the method comprising: Processing the input image by using a main neural network, the main neural network comprising an input layer, an output layer, and a plurality of hidden layers between the input layer and the output layer; While processing the input image through the main neural network to obtain a final output image, obtaining intermediate result data based on the input image by using the main neural network; as well as while the input image is being processed by using the main neural network to obtain the final output image, generating an intermediate image based on the intermediate result data output from one of the plurality of hidden layers of the main neural network by using a sub-neural network, Wherein, the sub-neural network includes one or more convolutional layers, and The method further includes: predicting a first time period taken to generate the final output image based on the input image and a second time period taken to generate the intermediate image based on the input image, and determining whether to generate the intermediate image by using the sub-neural network by comparing the difference between the predicted first time period and the second time period with a preset time.

9. The method according to claim 8, wherein: The main neural network includes a plurality of first convolutional layers as the plurality of hidden layers, Processing the input image by using the main neural network includes: extracting feature information by performing a first convolution operation on the input image and one or more kernels in the plurality of first convolutional layers, and Obtaining the intermediate result data includes: generating the intermediate result data based on the extracted feature information.

10. The method according to claim 8, wherein: Generating the intermediate image comprises: extracting feature information by applying the intermediate result data to one or more kernels in the one or more convolutional layers to perform a second convolution operation; and The intermediate image is generated based on the extracted feature information.

11. The method according to claim 8, further comprising: The final output image is generated by processing the input image using the main neural network.

12. The method according to claim 8, wherein: The main neural network is a network trained to generate the final output image by applying a preset pattern to the input image, and the degree to which the preset pattern is applied to the intermediate image is less than the degree to which the preset pattern is applied to the final output image.

Images