Apparatus and method for deep learning-based multi-exposure correction to improve real-time image exposure

Abstract

The present invention relates to a deep learning-based multiple exposure correction device, and may include a resolution converter capable of converting a high-resolution image into a low-resolution image, a luminance feature extractor capable of extracting luminance features from a low-resolution image, a detail feature extractor capable of enhancing detail features using a detail information enhancement technique and a plurality of normalization techniques, an exposure-based feature fusion unit capable of extracting exposure information based on the difference between a grayscale image of an input image and a predicted grayscale image and combining luminance features and detail features based thereon, and a dual-based upsampling unit capable of performing upsampling by estimating spatial information and intensity information based on high-resolution input information to minimize the spatial difference between a low-resolution image and a high-resolution image and to compensate for lost surrounding pixel information.

Description

Technology Field

[0001] The present invention relates to a deep learning-based multiple exposure correction device and method for improving real-time image exposure, and more specifically, to a deep learning-based multiple exposure correction device and method that accurately estimates appropriate brightness even in various exposure environments, simultaneously improves image detail and visual quality, and possesses real-time processing capabilities through an efficient computational structure. Background Technology

[0002] In existing image processing technologies, various approaches have been developed to address exposure issues. Underexposed or overexposed images not only significantly degrade visual quality but also cause performance degradation in computer vision tasks, such as object recognition, image classification, and video analysis, due to the loss of brightness and detail. To solve these problems, traditional techniques such as histogram equalization and gamma correction have been widely used. While histogram equalization improves overall exposure by adjusting the brightness distribution of an image, it has the disadvantage of losing image naturalness due to distortion of detail or excessive correction. Gamma correction is effective for adjusting specific brightness ranges, but its applicability to various exposure conditions has been limited.

[0003] Recently, deep learning-based exposure correction technology has been introduced, developing models capable of adapting to various exposure conditions by training on large datasets. This technology is useful for solving complex, non-linear exposure problems and offers the advantage of being applicable in a wider range of environments without being optimized for specific conditions. However, deep learning-based technology requires high computational costs during training and execution, and has limitations that make it difficult to apply in applications requiring real-time processing. In particular, in applications where real-time processing is essential, such as autonomous vehicles or drones, it becomes a major factor degrading system performance and efficiency. Furthermore, although this technology is capable of adapting to various exposure conditions, there have been reports of cases where detail restoration is insufficient in underexposed images, or where brightness adjustment is excessive in overexposed images, resulting in degraded image quality.

[0004] Existing technologies have clear limitations in terms of the accuracy and efficiency of exposure compensation, which hinders their usability in real-world application environments. These limitations are particularly pronounced in applications that require both real-time processing and high accuracy. Therefore, there is a need for a new technology capable of providing real-time processing performance without being constrained by exposure conditions. The problem to be solved

[0005] The present invention aims to significantly improve the potential for use in high-quality image restoration and real-time applications.

[0006] In addition, the present invention aims to enable high-quality image restoration while saving computational resources through a lightweight deep learning architecture.

[0007] However, the technical problem that this embodiment aims to solve is not limited to the technical problem described above, and other technical problems may exist. means of solving the problem

[0008] A deep learning-based multiple exposure correction device according to one embodiment of the present invention may include a resolution converter capable of converting a high-resolution image into a low-resolution image, a luminance feature extractor capable of extracting luminance features from a low-resolution image, a detail feature extractor capable of enhancing detail features using a detail information enhancement technique and a plurality of normalization techniques, an exposure-based feature fusion unit capable of extracting exposure information based on the difference between a grayscale image of an input image and a predicted grayscale image and combining luminance features and detail features based thereon, and a dual-based upsampling unit capable of performing upsampling by estimating spatial information and intensity information based on high-resolution input information to minimize the spatial difference between a low-resolution image and a high-resolution image and to compensate for lost surrounding pixel information.

[0009] According to one embodiment, the resolution converter can convert a high-resolution image into a low-resolution image through resampling or resolution reduction.

[0010] According to one embodiment, a luminance feature extractor can predict a corrected grayscale image and extract luminance features having a global brightness distribution.

[0011] According to one embodiment, the corrected grayscale image can be predicted through a convolution layer.

[0012] According to one embodiment, the detail information enhancement technique can enhance high-frequency components of an image using a Laplacian filter.

[0013] According to one embodiment, a plurality of normalization techniques may utilize one or more of instance normalization, layer normalization, and batch normalization.

[0014] According to one embodiment, the dual-based upsampling unit may include a spatial-based upsampling unit that performs high-resolution restoration based on the difference between the original image and a value obtained by converting the low-resolution image to the size of the original image, and an intensity-based upsampling unit that can reflect sufficient color information by converting the pixel intensity of the image through dimension expansion of the low-resolution image.

[0015] A deep learning-based multiple exposure correction method according to another embodiment of the present invention may include the steps of converting a high-resolution image into a low-resolution image, extracting luminance features from the low-resolution image, enhancing the detail features of the low-resolution image using a detail information enhancement technique and a plurality of normalization techniques, extracting exposure information based on the difference between the grayscale image of the input image and the predicted grayscale image, combining luminance features and detail features based on the extracted exposure information, and performing upsampling by estimating spatial information and intensity information, respectively, based on high-resolution input information to minimize the spatial difference between the low-resolution image and the high-resolution image and to compensate for lost surrounding pixel information.

[0016] According to one embodiment, the step of converting a high-resolution image into a low-resolution image may include the step of converting a high-resolution image into a low-resolution image through resampling or resolution reduction.

[0017] According to one embodiment, the luminance feature extraction step may include a step of predicting a corrected grayscale image and extracting luminance features having a global brightness distribution.

[0018] According to one embodiment, the detail information enhancement technique may include a step of enhancing high-frequency components of an image using a Laplacian filter.

[0019] According to one embodiment, a plurality of normalization techniques may include a step of using one or more of instance normalization, layer normalization, and batch normalization.

[0020] According to one embodiment, the step of performing upsampling may include a spatial-based upsampling step that performs high-resolution restoration based on the difference between the original image and a value obtained by converting the low-resolution image to the same size as the original image, and an intensity-based upsampling step that reflects sufficient color information by converting the pixel intensity of the image through dimension expansion of the low-resolution image. Effects of the invention

[0021] According to the present invention, complex image exposure differences can be corrected in real time and luminance and detail information can be effectively integrated to generate high-quality images.

[0022] In addition, according to the present invention, a lightweight deep learning-based image processing technology that can be utilized in various application environments while saving computational resources can be provided. Brief explanation of the drawing

[0023] Figure 1 is a diagram showing the process of generating a high-resolution restored image from an input image by a real-time image processing device in steps. Figure 2 is a diagram showing the configuration of a luminance feature extractor. Figure 3 is a diagram showing a dual-based upsampling configuration. Figure 4 is a diagram showing the results of comparing the performance of various low-light image enhancement and multiple exposure correction techniques using PSNR and SSIM indices. Specific details for implementing the invention

[0024] Embodiments of the present invention are described in detail below with reference to the attached drawings so that those skilled in the art can easily implement the invention. Since the present invention is susceptible to various modifications and may have various embodiments, specific embodiments are illustrated in the drawings and described in detail in the description. However, this is not intended to limit the present invention to specific embodiments, and it should be understood that it includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention.

[0025] To clearly explain the present invention, parts unrelated to the description have been omitted from the drawings, and similar parts throughout the specification have been given similar reference numerals. Furthermore, while describing with reference to the drawings, even components indicated by the same name may have different drawing numbers depending on the drawing, and drawing numbers are provided merely for the convenience of explanation; the concept, feature, function, or effect of each component is not to be interpreted restrictively by the corresponding drawing number.

[0026] Similar reference numerals are used for similar components when describing each drawing. Terms such as "first," "second," etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0027] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which this invention pertains.

[0028] Terms such as those defined in commonly used dictionaries should be interpreted as having meanings consistent with their meanings in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0029] Throughout the specification, when a part is described as being "connected" to another part, this includes not only cases where they are "directly connected" but also cases where they are "electrically connected" with other elements interposed between them. Furthermore, when a part is described as "comprising" a certain component, this means that, unless specifically stated otherwise, it does not exclude other components but rather allows for the inclusion of additional components; it should be understood that this does not preclude the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0030] In this specification, the term 'part' or 'module' includes a unit realized by hardware or software, or a unit realized using both; a single unit may be realized using two or more pieces of hardware, or two or more units may be realized by a single piece of hardware.

[0031] Hereinafter, a real-time multiple exposure correction device and method according to an embodiment of the present invention will be described with reference to FIGS. 1 to 4.

[0032] Figure 1 is a diagram showing the process of generating a high-resolution restored image from an input image by a real-time image processing device in steps.

[0033] Referring to FIG. 1, a real-time image processing device (100) may include a downsampling module (101), a Laplacian filter (LoG, 102), a convolution layer (103, 104), a Dual Feature Aggregation Module (DFAM, 105, 107), a multiple exposure correction device (106), and a dual-based upsampling unit (108).

[0034] First, image X is a low-resolution image (X) obtained through downsampling in the downsampling unit (101). l It is converted into ). Downsampling is a process that minimizes computation by reducing the spatial resolution of an image.

[0035] Generated low-resolution image (X l ) can be processed through two paths. The first path 1) is a structure designed to emphasize high-frequency components in input image data and extract detail information, in which a Laplacian filter (LoG, 102) is applied first. The Laplacian filter (102) improves overall data quality by removing high-frequency noise from the input image through Gaussian filtering while maintaining clear detail information, and can detect areas of rapid brightness change within the image by performing a Laplacian operation. In this process, edge and texture information are emphasized, and structural features of the image can be revealed more clearly.

[0036] The output data of the Laplacian filter (102) can be passed to a 3×3 convolution layer (103) for further processing. The 3×3 convolution layer (103) can further enhance the detail information by learning local detail information and high-frequency components in detail. That is, it can analyze the relationships between pixels within a small area (kernel or filter size) of the input data and extract important features from it. In this process, a small filter size (e.g., 3×3) can be used. The filter can explore a small area of ​​the input image and learn patterns that can detect high-frequency components such as boundaries, textures, and brightness changes, and emphasize them. For example, one filter can be used to detect horizontal boundaries, and another filter can be used to detect vertical boundaries or specific textures.

[0037] Path 2 represents a process for extracting global brightness and color information from a low-resolution image. In Path 2, input data passes through a separate 3×3 convolution layer (104), through which a basic feature map can be generated that captures the global structure and brightness patterns of the image. The generated feature map can be combined through DFAM (105). DFAM (105) is an advanced module that integrates different frequency bands and spatial information learned from multiple input paths to generate a high-level integrated feature map.

[0038] The operation of DFAM (105) begins by independently processing the feature maps transmitted from each input path. The first input path provides a feature map containing global spatial information and brightness information, while the second path provides a detail feature map with high-frequency components such as textures and edges emphasized. Since these input feature maps contain complementary information, DFAM (105) can process them independently and then combine them to generate an optimal integrated feature map.

[0039] The multiple exposure correction device (106) may include a Luminance Feature Extractor (LFE) (116), a Detail Feature Extractor (DFE) (126), and an exposure-based feature fusion device (136). The Luminance Feature Extractor (116) and the Detail Feature Extractor (126) can each extract luminance information and detail information. The Luminance Feature Extractor (116) can analyze the global brightness pattern of the image and generate a corrected grayscale image through 3×3 and 1×1 convolution operations and activation functions. Through this, brightness information is precisely adjusted. The Detail Feature Extractor can enhance high-frequency components such as textures and edges using a Laplacian filter and various normalization techniques, and can precisely extract detail information of the image. The detailed operation of the Luminance Feature Extractor (116) and the Detail Feature Extractor (126) will be explained in more detail through FIG. 2.

[0040] Subsequently, the exposure-based feature fusion unit (136) receives luminance information and detail information and can adaptively combine them according to exposure conditions. The exposure-based feature fusion unit (136) extracts exposure information by analyzing the difference between the input grayscale image and the predicted grayscale image, and can generate an optimized feature vector based on this. The detailed operation of the exposure-based feature fusion unit (136) will also be explained in more detail through FIG. 2.

[0041] The fused feature information can be further processed through the DFAM module (107) and transmitted with further improved precision. The DFAM module (107) can correct imbalances in the fused feature map generated in the previous step, optimize high-dimensional feature representations, and provide high-quality data that can be utilized in subsequent processing steps. The processing result of the DFAM (107) can be transmitted to the dual-based upsampling module (108) to finally be restored into a high-resolution image. The dual-based upsampling module (108) can use two complementary techniques in parallel: spatial-based upsampling and intensity-based upsampling.

[0042] In some embodiments, spatial-based upsampling, which is the first step of dual-based upsampling (108), can correct the size difference between the low-resolution image and the original image. This process can minimize distortion while expanding the size while preserving the spatial structure and pattern of the image. Spatial-based upsampling can map the pixel values ​​of the input image to a high-resolution coordinate system and then calculate missing pixel values ​​through an interpolation technique.

[0043] Intensity-based upsampling can restore color and brightness information. In this step, pixel intensity values ​​of low-resolution images are analyzed, and based on this, the color information required for high resolution can be supplemented. Intensity-based upsampling can restore visual quality by estimating lost surrounding pixel values.

[0044] In some embodiments, spatial-based upsampling and intensity-based upsampling are performed in parallel, thereby allowing the dual-based upsampling module (108) to prevent quality degradation that may occur during the high-resolution image restoration process. The dual-based upsampling module (108) simultaneously restores the spatial structure and color information of the input data and can ultimately generate a high-quality high-resolution image. The restored high-resolution image maintains both the detail information and global quality of the input image and can provide a visually improved result.

[0045] Figure 2 is a diagram showing the configuration of a luminance feature extractor (116).

[0046] Referring to FIG. 2, a luminance feature extractor (116) can extract luminance information from an input image and generate a corrected grayscale image. First, the input image passes through a 3×3 convolution layer (201). This convolution layer (201) can generate an initial luminance feature map by learning local brightness changes of the input image. The convolution operation performs a convolution operation between pixel values ​​and a learned filter, and can effectively extract textures and brightness patterns within the image. At this time, the 3×3 filter is used for learning local features and can be designed to handle low-resolution characteristics of the input data.

[0047] Next, the output of the convolution layer (201) is passed to a GELU (Gaussian Error Linear Unit) activation function (202). The GELU activation function (202) can perform smooth and continuous non-linear transformations by utilizing the probabilistic characteristics of the input values. This allows for more detailed learning of complex luminance patterns by adjusting the activation status according to the magnitude of the input values ​​without completely removing negative values. The outputs passed through the Conv (3x3) and GELU activation function can be combined with the input data through an addition operation. This process can strengthen the interaction between the original data and the learned data by adding the processed data while maintaining the original information of the input data.

[0048] The combined data is passed to a 3×3 convolution layer (203). The 3×3 convolution layer (203) can be used to capture complex brightness variations in the input data and enhance high-frequency information to generate a rich luminance feature map that includes both the overall brightness pattern and local details of the input image. Additionally, a GELU activation function (204) can be used to introduce nonlinearity to improve the expressiveness of the feature map.

[0049] Additionally, the activated luminance feature map can be passed to a 1×1 convolution layer (205). The 1×1 convolution layer (205) can compress multidimensional data to reduce computational load while preserving important luminance information. The 1×1 convolution (205) can selectively highlight important information in the high-dimensional feature map, and consequently provide a simplified data representation. This output is passed to a sigmoid activation function (206), and the sigmoid function (206) normalizes the output value to between 0 and 1 to obtain a corrected grayscale image ( Can generate ).

[0050] Finally, the generated corrected black and white image ( ) is a luminance-based loss function( It can be evaluated by the loss function( The difference between the predicted grayscale image and the original low-resolution image can be calculated and used as feedback for model training. Through this, the weights of the luminance feature extractor (116) are optimized and can be continuously improved to more accurately reproduce the brightness information of the input data. As the loss value becomes smaller, the generated result becomes more similar to the input image, and its utility in subsequent processing can be increased.

[0051] Next, the detail feature extractor (126) can extract and enhance detail information from the input image to precisely represent high-frequency components, textures, and boundary information. The detail feature extractor (126) is composed of a multi-convolution layer, a normalization layer, and a feature fusion operation, and operates stepwise to finally produce a refined detail feature map ( Creates ).

[0052] Initial detailed feature map ( The input data undergoes instance normalization (IN, 207). This normalization process can maintain data consistency and increase learning stability by adjusting the value range of the input data. The normalized feature map is passed to a 1×1 convolution layer (208), which can improve computational efficiency by reducing the number of channels of the data and compressing multidimensional data. Subsequently, a 3×3 convolution layer (209) can precisely learn the high-frequency components of the input data to enhance important details such as textures and boundaries. Finally, another 1×1 convolution layer (210) is additionally applied to remove unnecessary information from the data and preserve only optimized details. The output of the 1×1 convolution layer (210) is combined with information that has passed through a Laplacian filter (211) and a 3×3 convolution layer (212).

[0053] Next, the normalization layers, instance normalization (213), layer normalization (214), and batch normalization (214), are applied to normalize various characteristics of the input data. Instance normalization (213) performs normalization independently on each instance (image or sample) of the input data and can enhance the texture and pattern information of the data. Instance normalization (213) can contribute to restoring the texture of the image while reducing the influence of style information and preserving detail information. Layer normalization (214) performs normalization by calculating the mean and standard deviation at the level of all layers of the input data and can be used to stabilize structural data that is sensitive to changes in the scale (size) and distribution of the input data. Batch normalization (215) adjusts the mean and variance on a batch basis, which can reduce changes in the input data during training and improve the generalization performance of the model. Batch normalization (215) can increase the training speed and maintain data consistency, especially when the batch size is large.

[0054] Each normalization layer (213, 214, 215) processes the same input data in parallel to generate a respective normalization result, and these results can ultimately be integrated during the fusion process. All three normalized feature maps can be combined through a weighted sum operation (216). The weighted sum operation (216) reflects the relative importance of the data generated in each normalization layer, so that important information is emphasized and less important information is suppressed. The weights are dynamically adjusted during the learning process and can be determined according to the characteristics of the input data and the learning conditions.

[0055] The parallel normalization layers (213, 214, 215) have the advantage of being able to process various attributes of the input data in a balanced manner. Instance normalization (213) preserves the texture of the image precisely, layer normalization (214) stabilizes the global structure of the data, and batch normalization (215) can increase the efficiency of batch-based learning. This parallel normalization improves the overall quality of the input data and can increase the usability of the data in subsequent processing steps.

[0056] Subsequently, channel concatenation (217) operations are performed so that various feature maps can be merged into a single integrated data. The combined data sequentially passes through a DC 3×3 convolution layer (218) and a 1×1 convolution layer (219), and finally, a refined detail feature map ( Can generate ).

[0057] The exposure-based feature fusion unit (136) is a luminance feature map ( ) and detailed feature map( ) takes ) as input and outputs the final feature map( Can generate ).

[0058] First, the luminance feature map ( ) and detailed feature map( Each passes through layer normalization (LN, 220, 221). Layer normalization (217, 218) calculates the mean and variance in each feature map and normalizes them to maintain the stability of the learning and minimize the scale difference between input features.

[0059] The normalized luminance features and detail features are then combined through the Concatenate (222) operation to form a single feature map ( It can be combined into ). The Concatenate operation (222) enables interaction between luminance and detail by merging the channels of each input to add a new dimension. This combined single feature map ( ) integrates luminance and detail information so that each feature can complement each other.

[0060] Combined single feature map ( ) can reduce the spatial size through global average pooling (GAP, 223) and extract global statistical information for each channel. GAP (223) calculates the average value for each channel to represent the spatial information in the form of a compressed vector, which is then converted into an exposure-based emphasis value (e) learned by a multilayer perceptron (MLP, 224). The MLP (224) includes a non-linear activation function and can learn the complex relationships of the input data to generate appropriate weights under specific exposure conditions.

[0061] The generated highlight value (e) is expanded to its original spatial size through a broadcast (Broadcast, 225) operation, and the combined single feature map ( By performing multiplication operations with ), exposure information can dynamically adjust the importance of each channel. This process can amplify or suppress the contribution of each channel, making specific features more prominent. At the same time, the detail feature map ( ) is added to the combination process via a separate broadcast (225) path, which can complement new features while preserving original detail information.

[0062] Finally, it passes through Conv(1×1)(226) to adjust the interaction between channels and outputs a feature map ( It can generate ). Conv(1×1) adjusts the relationships between channels of the input data while maintaining spatial size, helping the network learn richer and more meaningful representations. The generated output feature map ( ) is the result of integrating exposure and detail information of input data, and can be used for more sophisticated feature learning and analysis in subsequent stages.

[0063] Figure 3 is a diagram showing a dual-based upsampling configuration.

[0064] Referring to FIG. 3, the dual-based upsampling (108) includes two main modules called Bilateral Guided Upsampling (310) and Detail Guided Upsampling (320), each of which can extract luminance and detail information from an input image and use them to generate an upsampled high-resolution image.

[0065] First, the input image (X) passes through a 3×3 convolution layer (301). The 3×3 convolution layer (301) learns the spatial relationships of the input image and can extract local luminance information. Afterwards, it can be normalized through a sigmoid activation function (302). The normalization process limits the range of luminance values ​​from 0 to 1, thereby increasing the stability of subsequent operations and improving the convergence speed of the model. The luminance map (Gi) prepared in this way can be used as input to the Bilateral Guided Upsampling module (310).

[0066] In the Bilateral Guided Upsampling module (310), a normalized luminance map (Gi) and an additional 3D luminance map (S) are combined to produce high-resolution luminance features ( ) is generated. S represents the spatial and luminance structure of the input image and can be combined with Gi and interpolated through the Trilinear Interpolation (304) process. Trilinear Interpolation (304) calculates the value of each pixel using surrounding 3D spatial data and can extend the luminance information of the input image to high resolution.

[0067] Upsampled input image ( ) passes through a 3×3 convolution layer (307). The generated Gs passes through a sigmoid activation function (308), and the normalized Gs is passed as input to the Detail Guided Upsampling module (320).

[0068] The Detail Guided Upsampling module (320) can perform Bilinear Interpolation (311) based on the detail map (Gs). In this process, Gs interpolates the detail value of each pixel using data from surrounding pixels, and as a result, a high-resolution detail map ( Bilinear Interpolation (311) generates ) and is relatively simple but effectively expands detail information and can smoothly express fine changes in detail.

[0069] Luminance-based features generated from Bilateral Guided Upsampling (310) ) and detail-based features generated from Detail Guided Upsampling (320) ) can be combined into a single feature map through a Concatenate operation (313). The Concatenate operation (313) combines the channels of the two feature maps so that luminance and detail information can work complementarily. The combined feature map passes through a Conv 1×1 layer (314) to adjust the interaction between channels and generate a final output map.

[0070] Finally, the generated output feature map can be combined with the input image X to be converted into an upsampled final result image.

[0071] Figure 4 is a figure showing the results of comparing the performance of various low-light image enhancement and multi-exposure correction techniques using PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity Index) indicators.

[0072] Referring to Fig. 4, the present invention achieved the highest values ​​in both major indicators, PSNR and SSIM, in a performance evaluation against a comparison model. PSNR is an indicator for evaluating the quality of a restored image, and a higher value indicates higher similarity to the original image and less distortion and noise. SSIM is an indicator that quantifies the structural similarity between the original image and the restored image, and a high SSIM indicates that the restored image preserves the detailed structure of the original image well. The present invention shows a PSNR of 24.00 and a SSIM of 0.860, surpassing technologies such as RetinexFormer (PSNR 23.34, SSIM 0.848) and CoTF (PSNR 23.89, SSIM 0.858).

[0073] Another feature of the present invention is the lightweighting of the model. With 0.08 million parameters, it has the smallest size among the comparison models. For example, while RetinexFormer has 1.60 million parameters and SID-ENC has 7.45 million, the present invention achieves equivalent or better performance with approximately 20 to 90 times fewer parameters. This significantly improves the efficiency of computational resources and enhances feasibility in real-time systems or mobile environments where memory and computational limits are present. This lightweighting demonstrates that innovative technologies were applied during the model design and training processes, going beyond merely reducing the model size.

[0074] This invention was designed considering not only performance indicators such as PSNR and SSIM but also image restoration performance in various environments. For example, it demonstrates high performance even in complex tasks such as image restoration in low-light environments and correction of multi-exposure images, suggesting practical applicability. In particular, compared to existing models designed for lightweighting, such as Zero-DCE (PSNR 18.96, SSIM 0.774), it provides overwhelmingly higher performance at the same model size.

[0075] The present invention holds an advantage in terms of both efficiency and quality, not only in terms of performance indicators such as PSNR and SSIM, but also when compared to modern models such as RetinexFormer and CoTF. Although RetinexFormer demonstrates high performance, its parameter count is very large at 1.60M, and CoTF also has a parameter count of 0.31M, which is still more than four times larger than the present invention.

[0076] The hardware architecture described above may be implemented as hardware components, software components, and / or a combination of hardware and software components. For example, the devices and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include multiple processing elements and / or multiple types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0077] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0078] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable storage medium. It is not limited to a medium directly connected to a computer system, but may be distributed over a network. Examples of media may include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and media configured to store program instructions, including ROM, RAM, and flash memory. Additionally, other examples of media may include recording media or storage media managed by app stores that distribute applications or sites and servers that supply or distribute various other software.

[0079] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0080] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below. Explanation of the symbols

[0081] 100: Real-time video processing device 101: Downsampling Module 102, 207, 211: Laplacian Filter (LoG) 103, 104, 201, 209, 212, 301, 307: 3×3 convolution layer 105, 107: DFAM 106: Multiple Exposure Compensation Device 108: Dual-based upsampling section 116: Luminance Feature Extractor 126: Detail Feature Extractor 136: Exposure-based feature fusion device 202: GELU Activation Function 203: 3×3 Convolution Layer 204: GELU activation function 205: 1×1 Convolution Layer 206, 302, 308: Sigmoid activation function 208, 210, 219, 226, 306, 314: 1×1 convolution layer 213, 220, 221: Instance Normalization 214: Layer Normalization 215: Batch Normalization 216: Weight Sum 217, 222, 313: Concat operation 223: GAP (Global Average Pooling) 224: MLP (Multilayer Perceptron) 225: Broadcast operation 304: Trilinear Interpolation 310: Bilinear Interpolation

Claims

Claim 1 A deep learning-based multi-exposure correction device for improving real-time video exposure comprises: a resolution converter that downsamples a high-resolution image provided as an input image and converts it into a low-resolution image having a lower resolution than the high-resolution image; a luminance feature extractor that generates a luminance feature map from the low-resolution image and predicts a corrected grayscale image; a detail feature extractor that applies a Laplacian filter to the low-resolution image to emphasize high-frequency components, generates normalized feature maps through a plurality of normalization layers, and generates a detail feature map using the result of combining the normalized feature maps; and an exposure-based feature fusion unit that receives the luminance feature map and the detail feature map as inputs, normalizes them respectively to generate a normalized luminance feature map and a normalized detail feature map, combines the normalized luminance feature map and the normalized detail feature map to generate a single feature map, performs global average pooling and a multilayer perceptron on the single feature map to generate an exposure-based emphasis value, and generates an output feature map by adjusting the channel importance of the single feature map based on the exposure-based emphasis value. A deep learning-based multiple exposure correction device comprising: a bilateral guided upsampling module that generates a normalized luminance map from the high-resolution image and generates high-resolution luminance features by interpolating the normalized luminance map and a 3D luminance map representing the spatial and luminance structure of the high-resolution image; a detail guided upsampling module that generates a detail map from an image generated by upsampling the low-resolution image to the size of the high-resolution image and generates high-resolution detail-based features by interpolating the detail map; and a dual-based upsampling unit that generates a final result image by combining the high-resolution luminance features and the high-resolution detail-based features. Claim 2 delete Claim 3 A deep learning-based multiple exposure correction device according to claim 1, wherein the luminance feature extractor predicts a corrected grayscale image and generates a luminance feature map having a global brightness distribution. Claim 4 A deep learning-based multi-exposure correction device according to claim 3, characterized in that the corrected grayscale image is predicted through a convolution layer. Claim 5 delete Claim 6 A deep learning-based multiple exposure correction device according to claim 1, wherein the plurality of normalization layers include an instance normalization layer, a layer normalization layer, and a batch normalization layer that process the same input data in parallel to generate the normalized feature maps. Claim 7 delete Claim 8 delete Claim 9 delete Claim 10 delete Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 delete

Images