Methods and systems for DNN-based imaging

Abstract

A method for performing DNN-based image processing by an imaging device is provided. The method comprises: obtaining at least one input image having multiple color channels by the imaging device; extracting semantic information from each color channel of the at least one input image and color correlation information from each color channel of the at least one input image by the imaging device at the same time; and generating at least one output image by the imaging device based on the extracted semantic information and the extracted color correlation information.

Description

Technical Field

[0001] The present disclosure relates to image processing, and more particularly, to a method and system for deep neural network (DNN) based imaging. Background Art

[0002] Running complex image processing applications on mobile platforms with limited computing power is important. Existing lightweight models for image classification and detection (e.g., MobileNet, ShuffleNet, etc.) are color information invariant and therefore not suitable for complex visual tasks such as low-light enhancement, denoising, dehazing, etc. In addition, existing lightweight models result in poor accuracy.

[0003] Existing heavyweight models (based on desktop computers) reuse the existing AlexNet and SqueezeNet architectures to provide comparable accuracy for vision applications, but are not suitable for computation on mobile platforms due to the large memory and computation requirements.

[0004] In addition, existing DNN architectures for imaging (based on mobile devices and desktops) are not universal. Different image use cases require different DNN architectures, which increases development and training time. Therefore, using existing DNN architectures is not efficient and is not compatible with mobile device platforms.

[0005] In addition, there is a growing demand for running deep convolutional neural network (DCNN) based applications on mobile platforms for imaging. However, mobile platforms have limited computing power and minimal storage capacity. Therefore, there is a need for models that are lightweight, fast, and general without sacrificing accuracy.

[0006] Color constancy is a fundamental block of the image processing pipeline and is inherently present in all vision-based applications, such as automatic white balance (AWB), semantic segmentation, and classification, etc. State-of-the-art methods for color constancy focus on improving accuracy, but at the expense of inference time and memory footprint.

[0007] Figure 1A is an example of an existing method with a neural network including multiple convolutional blocks for image processing. Figure 1A As shown, the input image passes through filter 1 and then the first convolution is performed. This process continues for N convolutions until the output prediction is produced. Due to the convolution operation in each block, this conventional method has very high computational complexity.

[0008] Figure 1B is an example of another conventional method having a neural network including decomposed convolutional blocks for image processing. Figure 1BAs shown, the input image passes through filter 1, then performs depthwise convolution, followed by pointwise convolution. Conventional methods can have low computational complexity for each convolution operation, but many layers need to be added to obtain higher accuracy, which significantly increases the computational time. Therefore, a lightweight and faster model without sacrificing accuracy is needed.

[0009] It would therefore be desirable to address the above-mentioned disadvantages or others, or at least provide a useful alternative. Summary of the invention

[0010] Therefore, the embodiments of the present invention disclose a DNN-based image processing method for an imaging device. The method includes: obtaining at least one input image having multiple color channels by the imaging device. The method also includes: extracting semantic information from each color channel by the imaging device at the same time, and extracting color correlation information from each color channel of the at least one input image. The method also includes: generating at least one output image by the imaging device based on the extracted semantic information and the extracted color correlation information. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The method and system are shown in the accompanying drawings, in which the same reference numerals represent corresponding parts in different drawings. The embodiments herein will be better understood from the following description with reference to the accompanying drawings, in which:

[0012] Figure 1A A block diagram showing image processing according to an embodiment disclosed in the prior art;

[0013] Figure 1B A block diagram showing image processing according to an embodiment disclosed in the prior art;

[0014] Figure 2 A block diagram of an imaging device for DNN-based training according to an embodiment is shown;

[0015] Figure 3A is a block diagram of a processor for DNN-based training according to an embodiment;

[0016] Figure 3B A schematic diagram of DNN-based imaging according to an embodiment is shown;

[0017] Figure 4A A block diagram showing the operation of a fusion block according to an embodiment is shown;

[0018] Figure 4B A schematic diagram showing the operation of a fusion block according to an embodiment is shown;

[0019] Figure 5A A schematic diagram showing semantic data in an input image according to an embodiment;

[0020] Figure 5B A schematic diagram showing color correlation data in an input image according to an embodiment;

[0021] Fig. 6A A flow chart of a method for performing DNN-based training on an imaging device according to an embodiment is shown;

[0022] Figure 6B is a flow chart illustrating a method for an imaging device to perform DNN-based image processing on an embodiment;

[0023] Figure 7 A schematic diagram of a method for converting any given CNN-based general model into the proposed DNN-based imaging device according to an embodiment is shown;

[0024] Figure 8 A flowchart of a method for converting any given CNN-based general model into the proposed DNN-based imaging device according to an embodiment is shown;

[0025] Fig. 9 A diagram showing an exemplary fogged image according to an embodiment, a defogged image using a conventional method, and a defogged image using the proposed method;

[0026] Fig. 10A shows an exemplary input image to an imaging device according to an embodiment;

[0027] Fig. 10B shows an exemplary output image from an imaging device according to an embodiment;

[0028] Fig. 10C shows an exemplary input image to an imaging device according to an embodiment;

[0029] Fig. 10D shows an exemplary output image from an imaging device according to an embodiment;

[0030] Fig.10E shows an exemplary input image to an imaging device according to an embodiment;

[0031] Fig.10F shows an exemplary output image from an imaging device according to an embodiment;

[0032] Fig.11 A diagram showing an exemplary low-light input image and an enhanced output image using the proposed method according to one embodiment;

[0033] Fig.12 A diagram showing an exemplary image color correction by the proposed method according to an embodiment.

[0034] Fig.13 A diagram showing an architecture with two design blocks for achieving higher precision in image processing, according to an embodiment.

[0035] Fig.14 Diagrams showing exemplary input images captured using a camera and enhanced output images obtained using the proposed method according to an embodiment. DETAILED DESCRIPTION

[0036] Therefore, the embodiments herein disclose a DNN-based image processing method for an imaging device. The method includes: obtaining, by an imaging device, at least one input image having multiple color channels. The method also includes: extracting, by the imaging device, semantic information from each color channel at the same time, and extracting color correlation information from each color channel of at least one input image. The method also includes: generating, by the imaging device, at least one output image based on the extracted semantic information and the extracted color correlation information.

[0037] The method further includes: performing, by the imaging device, channel-by-channel fusion of the semantic information and the color correlation information. In an embodiment, extracting the semantic information and the color correlation information includes: extracting the semantic information and the color correlation information separately by using each separate neural network.

[0038] In an embodiment, extracting the semantic information comprises: extracting the semantic information by using depthwise convolution.

[0039] In an embodiment, extracting the color correlation information comprises extracting the color correlation information by using point-wise convolution.

[0040] In an embodiment, performing channel-by-channel fusion of the semantic information and the color correlation information further comprises: fusing each pixel from each channel of the semantic information with each pixel from each channel of the color correlation information. The performing further comprises: generating, by the imaging device, a learned mapping between the semantic information and the color correlation information for each of the channels.

[0041] In an embodiment, generating at least one output image based on channel-by-channel fusion includes: generating a prediction value based on channel-by-channel fusion of semantic information and color correlation information by an imaging device. Obtaining also includes: correcting at least one input image based on the prediction value generated from the channel-by-channel fusion operation by the imaging device, and generating at least one output image based on the correction.

[0042] Therefore, the embodiments of the present invention disclose an imaging device for DNN-based training, which includes a memory, a communicator communicatively connected to the memory, and a processor connected to the memory. The processor can be configured to receive at least one input image having multiple color channels. The processor can also be configured to simultaneously extract semantic information from each color channel of at least one image and extract color correlation information from each color channel of at least one image. The processor can also be configured to perform channel-by-channel fusion of semantic information and color correlation information, and generate at least one output image based on the channel-by-channel fusion.

[0043] When considered in conjunction with the following description and the accompanying drawings, these and other aspects of the embodiments herein will be better understood and grasped. However, it should be understood that the following description, although indicating preferred embodiments and many specific details thereof, is provided by way of illustration and not limitation. Many changes and modifications may be made within the scope of the embodiments herein without departing from the spirit of the present invention, and the embodiments herein include all such modifications.

[0044] The embodiments herein and their various features and advantageous details will be explained more fully with reference to the non-limiting embodiments shown in the accompanying drawings and described in detail in the following description. Descriptions of well-known components and processing techniques are omitted so as not to unnecessarily make the embodiments herein unclear. In addition, the various embodiments herein are not necessarily mutually exclusive, as some embodiments can be combined with one or more other embodiments to form new embodiments. Unless otherwise stated, the term "or" used herein refers to a non-exclusive "or". The examples used herein are intended only to help understand the ways in which the embodiments herein can be practiced, and further enable those skilled in the art to practice the embodiments herein. Therefore, the embodiments should not be interpreted as limiting the scope of the embodiments herein.

[0045] As conventional in the art, embodiments can be described and illustrated according to blocks that perform one or more functions described. These blocks may be referred to herein as managers, units, modules, hardware components, etc., which are physically implemented by analog and / or digital circuits, such as logic gates, integrated circuits, microprocessors, microcontrollers, memory circuits, passive electronic components, active electronic components, optical components, hard-wired circuits, etc., and may be optionally driven by firmware and software. For example, a circuit may be implemented in one or more semiconductor chips, or implemented on a substrate support such as a printed circuit board. The circuits constituting the blocks may be implemented by dedicated hardware, or by a processor (e.g., one or more programmed microprocessors and associated circuits), or by a combination of dedicated hardware for executing some functions of a block and a processor for executing other functions of a block. Without departing from the scope of the present disclosure, each block of the embodiment may be physically divided into two or more interactive and discrete blocks. Similarly, without departing from the scope of the present disclosure, the blocks of the embodiment may be physically combined into more complex blocks.

[0046] The accompanying drawings are used to help easily understand various technical features, and it should be understood that the embodiments presented here are not limited by the accompanying drawings. Therefore, the present disclosure should be interpreted as extending to any changes, equivalents and substitutions in addition to those specifically set forth in the accompanying drawings. Although the terms "first", "second", etc. can be used to describe various elements in this article, these elements should not be limited by these terms. These terms are usually only used to distinguish one element from another element.

[0047] Therefore, the embodiments herein disclose a method for a DNN-based training method for an imaging device. The method may include: receiving, by the imaging device, at least one input image having multiple color channels. The method may also include: extracting, by the imaging device, semantic information from each color channel of at least one image simultaneously using deep convolution, and extracting color correlation information from each color channel of at least one image using point-by-point convolution. The method also includes: performing, by the imaging device, channel-by-channel fusion of semantic information and color correlation information. The method may also include: obtaining, by the imaging device, at least one output image based on channel-by-channel fusion.

[0048] In an embodiment, a DNN-based training method for an imaging device is provided. In an embodiment, semantic information and color correlation information are extracted from each color channel simultaneously. In an embodiment, a channel-by-channel fusion of the semantic information and the color correlation information is performed. In an embodiment, at least one output image is obtained based on the channel-by-channel fusion.

[0049] In another embodiment, a learned mapping between semantic information and color correlation information is generated for each channel.

[0050] In another embodiment, image components are predicted based on a channel-by-channel fusion of semantic information and color correlation information.

[0051] In another embodiment, at least one input image is corrected based on the predicted image components.

[0052] Referring now to the drawings, and more particularly to Figure 2 10 , wherein like reference numerals indicate corresponding features consistently throughout the drawings, a preferred embodiment is shown.

[0053] Figure 2 is a block diagram of an imaging device 200 for DNN-based training according to an embodiment disclosed herein. The imaging device 200 may be, for example but not limited to, a cellular phone, a tablet, a smart phone, a laptop, a personal digital assistant (PDA), a global positioning system, a multimedia device, a video device, an Internet of Things (IoT) device, a smart watch, a game console, etc. The imaging device 200 may also be referred to as a mobile station, a subscriber station, a mobile unit, a subscriber unit, a wireless unit, a remote unit, a mobile device, a wireless device, a wireless communication device, a mobile subscriber station, an access terminal, a mobile terminal, a wireless terminal, a remote terminal, a mobile phone, a user agent, a mobile client, etc. by those skilled in the art.

[0054] In an embodiment, the imaging device 200 includes a processor 210 , a memory 220 , and a communicator 230 .

[0055] The processor 210 is configured to execute instructions stored in the memory 220 and perform various processes. The processor 210 is also configured to receive an input image having multiple color channels. The processor 210 is also configured to simultaneously extract semantic information from each color channel of the input image using depthwise convolution, and extract color correlation information from each color channel of at least one image using pointwise convolution. The processor 210 is also configured to perform channel-by-channel fusion on the semantic information and the color correlation information, and obtain at least one output image based on the channel-by-channel fusion.

[0056] The communicator 230 is configured for internal communication between internal hardware components and with external devices via one or more networks.

[0057] Memory 220 stores effects and conditions. Memory 220 also stores instructions to be executed by processor 210. Memory 220 may include non-volatile storage elements. Examples of such non-volatile storage elements may include magnetic hard disks, optical disks, floppy disks, flash memory, or electrically programmable memory (EPROM) or electrically erasable programmable (EEPROM) memory. In addition, in some examples, memory 220 may be considered a non-transitory storage medium. The term "non-transitory" may mean that the storage medium is not included in a carrier or propagation signal. However, the term "non-transitory" should not be interpreted as memory 220 being non-removable. In some examples, memory 220 may be configured to store a larger amount of information than memory. In some examples, a non-transitory storage medium may store data that may change over time (e.g., in a random access memory (RAM) or cache memory).

[0058] although Figure 2 Various hardware components of the imaging device 200 are shown, but it should be understood that other embodiments are not limited thereto. In other embodiments, the imaging device 200 may include fewer or more components. In addition, the labels or names of the components are only for illustrative purposes and do not limit the scope of the present invention. One or more components can be combined to perform the same or substantially similar functions as for processing PDCP operations in a wireless communication system.

[0059] Figure 3A is a block diagram of a processor 210 for DNN-based imaging. Figure 3A As shown, the processor 210 includes an input block 302, a semantic information extraction block 304, a color correlation extraction block 306, a subtask fusion block 308, an image post-processing block 310, and an output block 312. In an embodiment, the input block 302 receives a plurality of input images. The input image includes a plurality of color channels. In an exemplary embodiment, the color channel may be an RGB channel.

[0060] After receiving multiple input images, the semantic information extraction block 304 and the color correlation extraction block 306 simultaneously perform convolution operations on the input images. In an embodiment, the semantic information extraction block 304 performs a depthwise convolution operation on multiple channels of the input images. By performing the depthwise convolution operation, semantic information is extracted from multiple color channels of the input images. In another embodiment, the color correlation extraction block 306 performs a pointwise convolution operation on multiple channels of the input images. By performing the pointwise convolution operation, color correlation information is extracted from multiple color channels of the input images.

[0061] In addition, the subtask fusion block 308 performs channel-by-channel fusion of the semantic information and the color correlation information. In an embodiment, the subtask fusion block 308 performs a Hadamard transform between each pixel from each channel of the semantic information and each pixel from each channel of the color correlation information. In addition, the subtask fusion block 308 generates a learning mapping between the semantic information and the color correlation information of each channel based on the Hadamard transform. The learning mapping between the semantic information and the color correlation information based on each channel can be generated by other methods known in the art.

[0062] In an embodiment, the subtask fusion block 308 is implemented as a simple dot product of semantic and color correlation information. Due to data locality, the calculation mechanism of the dot product is very efficient. The subtask fusion block 308 helps the proposed imaging device 200 to obtain better accuracy than the conventional imaging device 200.

[0063] After the channel-by-channel fusion is performed by the subtask fusion block 308, the image post-processing block 310 predicts the image components based on the channel-by-channel fusion of the semantic information and the color correlation information. After predicting the image components, the output block 312 corrects at least one input image based on the predicted image components. In addition, the output block 312 obtains at least one output image based on the correction.

[0064] In an exemplary embodiment, in the case of color correction, the output from the subtask fusion block 308 is sent to the image post-processing block 310 to predict the luminance of the RGB components. In another embodiment, in the case where the input image has low light, the image post-processing block 310 generates a low light enhanced image.

[0065] Figure 3B A schematic diagram of DNN-based imaging according to an embodiment disclosed herein is shown.

[0066] In an exemplary embodiment, the processor 210 extracts the semantic information and the color correlation information separately by using each separate neural network. In an exemplary embodiment, each of the semantic information and the color correlation information is extracted from the input image 10 in parallel.

[0067] like Figure 3B As shown, in an exemplary embodiment, at 314, the processor 210 extracts semantic information from the input image 10 using a deep neural network including a deep convolution operation. In an exemplary embodiment, at 316, the processor 210 extracts color correlation information from the input image 10 using a deep neural network including a point-by-point convolution operation.

[0068] In an exemplary embodiment, at 318 , processor 210 performs a channel-by-channel fusion of the extracted semantic information and the extracted color correlation information.

[0069] In an exemplary embodiment, the processor 210 generates the output image 20 based on the extracted semantic information and the extracted color correlation information. The processor 210 generates the output image 20 based on the fusion of the semantic information and the color correlation information.

[0070] In an exemplary embodiment, the processor 210 generates the output image 20 based on separately extracting the semantic information and the color correlation information from the input image 10 in parallel.

[0071] In another exemplary embodiment, the processor 210 extracts another feature information from the input image. The processor 210 extracts at least two other feature information from the input image in parallel.

[0072] For example, the processor 210 extracts information about the distribution of pixel values ​​along the spatial dimension. Also, for example, the processor 210 extracts information about the distribution of pixel values ​​along the depth (channel) dimension. In an exemplary embodiment, learning the distribution of pixel values ​​across the space-time or depth dimension can provide an intensity histogram of the pixel. Moreover, learning the distribution of pixel values ​​across the space-time or depth dimension can provide color constancy.

[0073] For another example, the processor 210 extracts information about statistical regularities across spatial dimensions. In addition, the processor 210 extracts information about statistical regularities across depth (channel) dimensions. For example, the processor 210 extracts statistical properties such as mean, median, variance, skewness (asymmetry in the distribution of pixel values), kurtosis (intensity peaks in the distribution of pixel values). In an exemplary embodiment, learning kurtosis can provide the amount of noise present in the input image.

[0074] For example, another feature information extracted by the processor 210 from the input image may be information about capturing a dark channel feature. This feature can be used to defog the image. The dark channel of an image is defined as the minimum value of all pixel colors in a local slice of the image. In most local areas that do not cover the sky, some pixels (called "dark pixels") usually have very low intensity in at least one color (r, g, b) channel. For example, in a haze image, the intensity of these dark pixels in this channel is mainly contributed by air light. Therefore, these dark pixels can directly provide an accurate estimate of haze transmission. This can be performed in the first branch of the proposed architecture.

[0075] For example, the feature information can be information about the estimated atmospheric light feature. This feature can be used in image dehazing. The brightest pixels in the dark channel can be estimated. These pixels are mostly foggy and opaque. Among these pixels, the pixels with the highest intensity in the input image are selected as atmospheric light. This can be performed in the second branch of the proposed architecture.

[0076] For example, the feature information can be information about a salient feature in the image. The feature can be used to deblur the image. To detect the important parts in the image, the foreground can be segmented from the background. The blurred parts (background) and the non-blurred (foreground) parts in the image can be detected and classified. This can be performed in the first branch of the proposed architecture.

[0077] For example, the feature information can be information about motion vectors or optical flow estimates. This feature can be used to deblur the image. Motion vectors are used to determine optical flow, which estimates the motion and its direction. This is used to compensate for motion (object or camera motion) that causes blur in the image. This can be performed in the first branch of the proposed architecture.

[0078] For example, the feature information can be information about the estimated kurtosis. This feature can be used to denoise the image. By estimating the distribution of pixels present in the image and estimating the intensity peak in the distribution of pixel values, the amount of noise can be estimated and the noise present in the image can be modeled. This is performed in two branches of the proposed architecture.

[0079] For example, the feature information can be information about the pixel distribution. This feature can be used for color correction and low-light images. The pixel intensity distribution on the R, G, B channels in the image can be captured. It gives an estimate of whether the information present in the image is too bright or too dark. In addition, the histogram generated from the pixel distribution can be used to model the brightness and noise. This is performed in the first branch of the proposed architecture.

[0080] In an embodiment, the processor 210 generates the output image based on extracting at least two other feature information from the input image in parallel.

[0081] Figure 4A 3 is a schematic diagram illustrating the function of the subtask fusion block 308. Figure 4A As shown, at the semantic information extraction block 304, semantic information is extracted from multiple channels of the input image.

[0082] In an exemplary embodiment, after extracting semantic information and color correlation information, two output feature maps (a semantic feature map and a color feature map, each having "N" channels) are generated from the input image, for example.

[0083] Both output feature maps have multiple channels. For example, the semantic feature map (extracted from the first branch) has "N" channels. Similarly, the color feature map (extracted from the second branch) has "N" channels.

[0084] In an exemplary embodiment, for example, Figure 4A As shown, 402a and all pixels on the plane are channel 1 of semantic information. 402b and all pixels on the plane are channel 1 of color correlation information. Figure 4A As shown, 404a and all pixels on the plane are channel 2 of semantic information. 404b and all pixels on the plane are channel 2 of color correlation information.

[0085] 402a and 402b represent a first depth channel of a plurality of depth channels of an input image. In addition, 402a represents semantic information extracted from the first depth channel, and 402b represents color correlation information extracted from the first depth channel. 404a and 404b represent at least one second depth channel of a plurality of depth channels of an input image. In addition, 404a represents semantic information extracted from at least one second depth channel of a plurality of channels of an input image, and in addition, 404b represents color correlation information extracted from at least one second depth channel of a plurality of channels of an input image.

[0086] In an exemplary embodiment, 402a and 404a are semantic feature blocks for different pixel portions in their respective feature maps. 402b and 404b are color feature blocks for different pixel portions in their respective feature maps. As shown in the subtask fusion block 308, 402a and 402b are fused with a (+) operator to obtain a fused output at the corresponding portion. 404a and 404b in the second feature map are fused to obtain the corresponding portion in the subtask fusion block 308. Figure 4A The fusion operator (+) in the channel-by-channel fusion block can adopt multiple formulas, such as addition, multiplication, bitwise OR-OR-XOR, etc.

[0087] The subtask fusion block 308 fuses the extracted semantic information 402a with the extracted color correlation information 402b by performing a Hadamard transform. In an embodiment, the extracted semantic information 402a may be fused with the extracted color correlation information 402b by performing various techniques known in the art. In addition, the subtask fusion block 308 fuses the extracted semantic information 404a with the extracted color correlation information 404b. This process is repeated by the subtask fusion block 308 for multiple channels of the input image.

[0088] Figure 4B is a schematic diagram illustrating the operation of the fusion block.

[0089] like Figure 4B As shown, in an embodiment, the subtask fusion block 308 generates multiple learning mappings 406 between semantic information and color correlation information for each channel (channel 1, channel 2, channel N) based on Hadamard transform. The multiple learning mappings are fused together by the subtask fusion block 308 to generate an image component for at least one input image. In an embodiment, the subtask fusion block 308 can generate multiple learning mappings between semantic information and color correlation information for each channel based on techniques known in the prior art.

[0090] like Figure 4B As shown, in an embodiment, in order to apply semantic features (from branch 1) to color features (from branch 2), a merging technique called "channel-wise weighted pooling" can be used. The proposed channel-wise weighted pooling does not limit the number of weighted filters to 1, but provides a semantic filter for each color correlation information extracted. In an embodiment, at least one of the filters generated by the semantic information extraction block is applied to at least one of the color correlation information extracted from the color correlation extraction block.

[0091] Furthermore, the number of extracted color correlation information may be more than three, which is later reduced to match the required output channel dimension.

[0092] In an embodiment, one or more semantic filters corresponding to each depth channel are generated. One or more color correlation information corresponding to each depth channel is extracted. Each of the one or more semantic filters is fused with the color correlation information corresponding to each of the depth channels.

[0093] According to an embodiment, channel-by-channel multiplication may enable fine-tuning and result in better performance.

[0094] Figure 5A Schematic diagram of an exemplary input image including semantic information and color correlation information according to embodiments disclosed herein is shown. Image 502 is an exemplary input image. The semantic information in the image spans the entire image spatially, and the pixel color information spans the depth (R, G, B color channels) of the image. Figure 5A As shown, the foreground, background, existence of objects in the image, and relationships between multiple objects representing semantic information of the image 502 are determined.

[0095] Figure 5BSchematic diagram of an exemplary input image including color correlation information according to embodiments disclosed herein is shown. Image 504 is an exemplary input image. The pixel-by-pixel color information spans the depth (R, G, B color channels) of image 504. Figure 5B As shown, the relationship between the color values ​​in each channel is determined. In addition, the color correlation information also indicates the existence of independent pixels that characterize the contrast image related to the color correlation information of the image 504.

[0096] Fig. 6A 600 is a flowchart illustrating a method for performing DNN-based training on an imaging device according to an embodiment disclosed herein. At 602, the imaging device 200 receives at least one input image from a plurality of input images. After receiving the at least one input image, at 604, semantic information is extracted from the at least one input image by performing a deep convolution on the at least one input image. At 606, color correlation information is extracted from the at least one input image by performing a point-by-point convolution on the at least one input image. After receiving the input image, the imaging device 200 performs steps 604 and 606 simultaneously. By performing the deep convolution operation and the point-by-point convolution operation simultaneously, the computation is reduced by 70 times compared to conventional methods. In addition, the DNN model size is also reduced by 50 times compared to the prior art methods. Therefore, the proposed architecture is very suitable for real-time mobile vision applications.

[0097] After extracting semantic information and color correlation information from at least one input image, the flow proceeds to 608 .

[0098] At 608 , the imaging device 200 performs a channel-by-channel weighted fusion operation for obtaining image components of the at least one input image. In an embodiment, the imaging device 200 generates a multi-scale context map and sends it to a decoder in the image post-processing block 310 .

[0099] At 610 , the imaging device 200 decodes the multi-scale context maps and combines the multi-scale context maps to reconstruct at least one input image at the original resolution. At 612 , the imaging device 200 generates an output image based on the correction at 610 .

[0100] Figure 6B 601 is a flowchart illustrating a method of DNN-based image processing performed by an imaging device for embodiments as disclosed herein.

[0101] At 614, the imaging device 200 obtains an input image. In an embodiment, the imaging device 200 obtains at least one input image having multiple color channels.

[0102] For example, the imaging device 200 obtains at least one image using a camera included in the imaging device 200. For example, the imaging device 200 receives at least one image from an external device. For example, the imaging device 200 obtains at least one image stored in the memory 220 as an input image.

[0103] At 616, the imaging device 200 extracts semantic information from the input image and extracts color correlation information from the input image. In an embodiment, the imaging device 200 simultaneously extracts semantic information from each color channel of at least one input image and extracts color correlation information from each color channel of at least one input image.

[0104] In an embodiment, the imaging device 200 performs channel-by-channel fusion of semantic information and color correlation information.

[0105] At 618 , the imaging device 200 generates an output image based on the extracted semantic information and the extracted color correlation information.

[0106] In an embodiment, the imaging device 200 generates an output image based on a channel-by-channel fusion of semantic information and color correlation information.

[0107] Figure 7 is a schematic diagram illustrating a method for converting any given CNN-based general model into the proposed DNN-based imaging device 200.

[0108] like Figure 7 As shown in FIG. 702, 702 is an existing CNN model. However, the existing CNN model is not as efficient as the DNN-based model shown in FIG. Figure 7 As shown, 704 is metadata information corresponding to at least one input image. In addition, at box 706, a subtask selection module is determined. The subtask selection module determines and splits the CNN calculation to match the calculation of the proposed DNN-based model shown in Figure 3. At box 708, the automatic generation engine generates an efficient CNN model by using deep convolution operations, point-by-point convolution operations, and channel-by-channel weighted fusion blocks. Box 710 represents a SNAP framework for an efficient CNN model including various elements such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and a neural processing unit (NPU).

[0109] Figure 8800 is a method for converting any given general CNN-based model into the proposed DNN-based imaging device 200. At 802, the imaging device 200 receives a main visual task and an associated convolutional network structure containing repeated convolution blocks. In addition, at 804, the imaging device 200 receives meta-information of associated subtasks (e.g., semantic reasoning and low-level pixel-related reasoning) from the main task. At 806, the imaging device 200 determines the number of filters required for each subtask at each layer of the neural network and the association of deep convolution with point-by-point convolution based on the subtask. At 806, based on the subtask, the association of deep convolution with point-by-point convolution and the number of filters required for each subtask at each layer of the neural network are determined. At 808, a multi-branch neural network including only deep convolution or point-by-point convolution is generated by the imaging device 200. At 810, the outputs of the subtasks are combined using channel-by-channel weighted merging to generate a complete neural network architecture.

[0110] Furthermore, by leveraging the SNAP capability, additional acceleration can be achieved, which makes the proposed DNN model well suited for very high FPS real-time mobile vision applications, such as: video deblurring, video color correction, high FPS low-light video recording.

[0111] Fig. 9 Figure 2 shows an exemplary fogged image, a defogged image using a conventional method, and a defogged image using the proposed method. Fig. 9 As shown, 902 represents an exemplary fogged image, 904 is a defogging image using a conventional method, and 906 is a defogging image using the proposed method. The defogging image 904 is formed by an existing heavyweight DNN model. When running on a mobile platform, the existing DNN model faces frame loss due to more computational complexity, resulting in a poor user experience. In addition, the existing DNN model also occupies more memory and causes memory overhead.

[0112] The dehazed image 906 is formed by the proposed lightweight and faster DNN model. The proposed model does not face any frame loss when running on a mobile platform because the proposed method has very small computational complexity. In addition, the proposed method also occupies less memory, almost thirty times less than existing methods.

[0113] FIG. 10A to FIG. 10B Exemplary hazy images and dehazed images using the proposed method are shown, respectively. FIG. 10C to FIG. 10D Exemplary noisy images and denoised images using the proposed method are shown, respectively. Figures 10E to 10F Exemplary blurred images and deblurred images using the proposed method are shown, respectively.

[0114] Fig.11 Graphs showing exemplary low-light input images and enhanced output images using the proposed method according to embodiments disclosed herein. Fig.11 As shown, images 1102a, 1104a, and 1106a are low-light input images of the imaging device 200. Using the proposed method, images 1102b, 1104b, and 1106b are corresponding enhanced output images from the imaging device 200. As can be seen from images 1102b, 1104b, and 1106b, it should be understood that the proposed method converts very low-light images into natural images of optimal quality.

[0115] Fig.12 FIG. 4 shows a diagram of image color correction of an exemplary image by the proposed method according to an embodiment disclosed herein. Fig.12 As shown, 1202 is an input image of the imaging device 200. At 1204, the imaging device 200 learns the relevant color information on the input image 1202 by performing a point-by-point convolution operation on the input image 1202. At 1206, the imaging device 200 learns which pixels are important for estimating the illumination, and accordingly assigns priorities to them using a depth convolution operation on the input image. At 1208, the imaging device 200 uses a channel-by-channel weighted merging layer to fuse the outputs obtained from 1204 and 1206. At 1210, the imaging device 200 performs spatial reduction on the output obtained at 1208. In an embodiment, at 1210, the dimension of the output obtained at 1208 is reduced to match the dimension of the desired prediction. At 1212, the depth reduction output is obtained by subtracting the output color of 1210 from the input image 1202 to obtain a color-corrected image 1214.

[0116] Fig.13 A schematic diagram of an architecture with two design blocks for achieving higher precision according to embodiments disclosed herein is shown. Fig.13 The structure shown includes two design blocks for processing input images. Fig.13 As shown, for design A, a convolution block is used to perform a fusion operation of point-by-point convolution and depth-wise convolution. Design A performs convolution on the cascaded task-specific input. For design B, a multi-broadcast block is used to perform a fusion operation of point-by-point convolution and depth-wise convolution. Design B uses a channel-by-channel weighted merging method to combine input signals. In an embodiment, in design B, a Hadamard transform can be used to perform the fusion operation. Fig.13 The structure shown provides higher accuracy for processing the input image.

[0117] Fig.14Diagrams showing exemplary input images captured by using a camera and enhanced output images obtained using the proposed method according to embodiments disclosed herein.

[0118] In an embodiment, the imaging device 200 includes at least one camera (not shown). In an embodiment, the imaging device 200 also includes a display 240 .

[0119] like Fig.14 As shown, in an embodiment, the processor 210 controls at least one camera to capture an image based on user input. The processor 210 controls the display 240 to display the captured image 1401. For example, the captured image 1401 may be an image with haze.

[0120] In an embodiment, the processor 210 generates an image 1402 without haze from an image 1401 with haze using the proposed method. In an exemplary embodiment, the processor 210 obtains the captured image 1401 as an input image for image processing. The processor 210 generates an image 1402 as an output image, the haze of which is removed and the image quality of which is enhanced.

[0121] like Fig.14 As shown, the processor 210 controls the display 240 to display the image 1402 without haze.

[0122] The embodiments disclosed herein may be implemented using at least one software program that runs on at least one hardware device and performs network management functions to control the elements.

[0123] The foregoing description of specific embodiments will sufficiently reveal the general nature of the embodiments herein so that other embodiments may be easily modified and / or adapted to various applications by applying current knowledge without departing from the main concept, and therefore, such adaptations and modifications should and are intended to be understood to be within the meaning and scope of equivalents of the disclosed embodiments. It should be understood that the wording or terminology used herein is for descriptive purposes rather than for limiting purposes. Therefore, although the embodiments herein have been described according to preferred embodiments, those skilled in the art will recognize that the embodiments herein may be practiced by modification within the spirit and scope of the embodiments described herein.

Claims

1. A method for image processing based on DNN, wherein the image processing is performed by an imaging device, and the method comprises: Acquiring at least one input image having a plurality of color channels by the imaging device; extracting, by the imaging device and by a first branch, semantic information from each of the color channels of the at least one input image at a first resolution; extracting, by the imaging device and by a second branch, color correlation information from each of the color channels of the at least one input image at the first resolution; as well as At least one output image is generated by the imaging device based on the extracted semantic information and the extracted color correlation information.

2. The method of claim 1, further comprising: Channel-by-channel fusion of the semantic information and the color correlation information is performed by the imaging device.

3. The method of claim 2, wherein: The at least one input image has a plurality of depth channels defined along a depth dimension, The method further comprises: generating one or more semantic filters corresponding to each of the plurality of depth channels, and Wherein, performing the channel-by-channel fusion includes: fusing each of the one or more semantic filters with the extracted color correlation information corresponding to each of the plurality of depth channels.

4. The method of claim 1, wherein: Extracting the semantic information and the color correlation information includes extracting the semantic information and the color correlation information separately by using respective separate neural networks.

5. The method of claim 1, wherein: Extracting the semantic information includes extracting the semantic information by using depthwise convolution.

6. The method of claim 1, wherein: Extracting the color correlation information includes extracting the color correlation information by using point-by-point convolution.

7. The method of claim 2, wherein: Performing the channel-by-channel fusion of the semantic information and the color correlation information comprises: The imaging device fuses pixels from the channels of the semantic information and the color correlation information for each of the color channels; and A learned mapping between the semantic information and the color correlation information is generated by the imaging device for each of the color channels based on the fusion.

8. The method of claim 2, wherein: Generating the at least one output image comprises: Generating, by the imaging device, a prediction value based on the channel-by-channel fusion of the semantic information and the color correlation information; correcting, by the imaging device, the at least one input image based on the prediction values ​​generated from the channel-by-channel fusion; and The at least one output image is generated by the imaging device based on the correction.

9. An imaging device for DNN-based image processing, the imaging device comprising: Memory; a processor coupled to the memory and configured to: Obtaining at least one input image having a plurality of color channels; extracting semantic information from each of the color channels of the at least one input image at a first resolution by a first branch; extracting, by a second branch, color correlation information from each of the color channels of the at least one input image at the first resolution; as well as At least one output image is generated based on the extracted semantic information and the extracted color correlation information.

10. The imaging device according to claim 9, wherein The processor is further configured to perform channel-by-channel fusion of the semantic information and the color correlation information.

11. The imaging device according to claim 10, in, The at least one input image has a plurality of depth channels defined along a depth dimension; Wherein, the processor is further configured to: generate at least one semantic filter corresponding to each of the plurality of depth channels; and The processor is further configured to: fuse each of the at least one semantic filter with the extracted color correlation information corresponding to each of the plurality of depth channels.

12. The imaging device according to claim 9, wherein: The processor is configured to separately extract the semantic information and the color correlation information by using respective separate neural networks.

13. The imaging device according to claim 9, wherein: The processor is configured to extract the semantic information by using deep convolution.

14. The imaging device according to claim 9, wherein: The processor is configured to extract the color correlation information by using point-wise convolution.

15. A non-transitory computer-readable recording medium having an executable program recorded thereon, wherein: When executed by at least one processor, the program instructs a computer to perform the following operations: Acquiring at least one input image having a plurality of color channels by an imaging device; By the imaging device and through the first branch at a first resolution extracting semantic information from each of the color channels of the at least one input image; extracting, by the imaging device and by a second branch, color correlation information from each of the color channels of the at least one input image at the first resolution; as well as At least one output image is generated by the imaging device based on the extracted semantic information and the extracted color correlation information.

Images