Electronic device for performing image processing by combining frequency domain and spatial domain and operating method thereof

Abstract

An electronic device is disclosed that performs image processing by combining a frequency domain and a spatial domain. The electronic device comprises: at least one processor; and a memory that stores instructions instructing the at least one processor to perform at least one operation. The at least one operation includes: an operation of converting an input image into a frequency domain to obtain a first image; an operation of performing a first image processing on the first image to obtain a second image; an operation of inversely converting the second image to obtain a spatial domain-based third image; and an operation of performing a second image processing on the third image to obtain an output image.

Description

Technology Field

[0001] The present invention relates to an image processing technology for improving image quality, and more specifically, to an electronic device that performs image processing by combining the frequency domain and the spatial domain, and a method of operating the same. Background Technology

[0002] Image processing technology has been applied to various electronic devices, including cameras, displays, medical imaging equipment, security surveillance systems, mobile terminals, and autonomous driving sensor systems. Generally, image quality enhancement technologies—such as sharpness improvement, noise reduction, image quality enhancement, compressed image restoration, and super-resolution reconstruction—are broadly classified into spatial domain-based and frequency domain-based processing methods.

[0003] Spatial domain-based image processing technology is a method that improves image quality by utilizing mathematical relationships between each pixel and its neighbors based on the positional coordinates of the pixels. Representative techniques include sharpening filters, blurring, non-linear filtering (bilateral filtering, median filtering), and edge-based operations. Recently, AI and deep learning-based restoration techniques using Convolutional Neural Networks (CNNs) or Transformer-based structures have also been included in spatial domain processing. While these methods offer intuitive computational implementation and are advantageous for detecting the shapes or contours of specific objects, they can result in the loss of detailed contours or the occurrence of discontinuous artifacts in noisy images.

[0004] Meanwhile, frequency domain-based image processing technology improves image quality by converting pixel information of an image into frequency components using Fourier Transform, Discrete Cosine Transform (DCT), or Wavelet Transform, and then adjusting each frequency band. For example, the low-frequency region primarily contains overall brightness and structural information of the image, so it is used for noise reduction or removal of compression artifacts, while the high-frequency region contains detailed texture or edge information, so it is used for resolution enhancement or texture restoration. Although this method has the advantage of being able to precisely control image characteristics at the frequency component level, it has limitations such as increased computational complexity during the frequency conversion process and the difficulty of processing only specific regions of the image independently.

[0005] However, most conventional technologies rely on either a spatial domain-based or a frequency domain-based processing method, resulting in the following technical limitations.

[0006] First, the single-domain-based processing method results in performance variations depending on the characteristics of each image type.

[0007] Secondly, a trade-off occurs during the noise removal process where image details are lost.

[0008] Thirdly, artifacts such as ringing, halo, and overshoot may occur during the frequency amplification process. The problem to be solved

[0009] The objective of the present invention to solve the above-mentioned problems is to provide an electronic device that performs image processing by combining the frequency domain and the spatial domain, and a method of operation thereof. means of solving the problem

[0010] One aspect of the present invention for achieving the above objective provides an electronic device that performs image processing by combining a frequency domain and a spatial domain.

[0011] The electronic device comprises at least one processor; and a memory that stores instructions that instruct the at least one processor to perform at least one operation.

[0012] The above at least one operation includes: an operation of converting an input image into the frequency domain to obtain a first image; an operation of performing a first image processing on the first image to obtain a second image; an operation of performing an inverse frequency transform on the second image to obtain a third image based on the spatial domain; and an operation of performing a second image processing on the third image to obtain an output image.

[0013] The above first image processing is performed using a first artificial neural network based on deep learning that has been pre-trained using training data.

[0014] The above second image processing is performed using a deep learning-based second artificial neural network that has been pre-trained using the above training data.

[0015] The above training data consists of training input data and correct answer data, and the above correct answer data includes a spatial domain-based correct answer image collected in advance and a frequency domain-based frequency correct answer image obtained by frequency transforming the above correct answer image.

[0016] The above at least one operation further includes: an operation of calculating a first loss by comparing the difference between the second image and the frequency correct image; an operation of calculating a second loss based on the difference between the output image and the spatial domain-based correct image; and an operation of adjusting the weights of the first artificial neural network and the second artificial neural network using the first loss and the second loss.

[0017] The operation of adjusting the weights includes adjusting the weights of the first artificial neural network so that the first loss is minimized, and adjusting the weights of the second artificial neural network so that the second loss is minimized.

[0018] The training input image included in the above training input data is generated by artificially applying at least one image quality distortion or degradation treatment among noise, compression artifact, blur, and illumination imbalance to the above correct image.

[0019] The above training data is collected by classifying it according to environmental conditions including at least one of lighting environment, shooting conditions, weather conditions, playback conditions, and object type.

[0020] The above lighting environment includes at least one of an indoor environment, an outdoor environment, a natural light environment, an LED lighting environment, a night IR shooting environment, and a low-light lighting environment.

[0021] The above shooting conditions include at least one of shooting in a stationary state, shooting while moving, shooting shake, and shooting focus error.

[0022] The above weather conditions include at least one of fog, rain, snow, backlight, and smoke.

[0023] The above playback conditions include at least one of playback of real-time streaming video, playback of compressed video, and playback of video including packet loss.

[0024] The above object types include at least one of a person, vehicle, building, indoor structure, document image, or natural landscape among the objects included in the image.

[0025] The above at least one operation further comprises: an operation of configuring a learning dataset including at least one of a first learning data collected according to the lighting environment, a second learning data collected according to the shooting conditions, a third learning data collected according to the weather conditions, a fourth learning data collected according to the playback conditions, and a fifth learning data collected according to the object type; and an operation of storing at least one of a first setting value for the first artificial neural network and the second artificial neural network learned using the first learning data, a second setting value for the first artificial neural network and the second artificial neural network learned using the second learning data, a third setting value for the first artificial neural network and the second artificial neural network learned using the third learning data, a fourth setting value for the first artificial neural network and the second artificial neural network learned using the fourth learning data, and a fifth setting value for the first artificial neural network and the second artificial neural network learned using the fifth learning data.

[0026] The above at least one operation further includes: an operation to determine environmental conditions corresponding to the input image; an operation to determine a setting value corresponding to the determined environmental conditions; and an operation to set the first artificial neural network and the second artificial neural network using the determined setting value.

[0027] The operation of adjusting the weights includes: an operation of calculating an integrated loss using the first loss and the second loss; and an operation of adjusting the weights of the first artificial neural network and the second artificial neural network by performing a backpropagation algorithm so as to minimize the integrated loss.

[0028] The above at least one operation further includes: an operation of calculating an image quality index by comparing the output image and the correct image; and an operation of adjusting the weights of the first artificial neural network and the second artificial neural network based on the calculated image quality index.

[0029] The above image quality indicators include at least one of SSIM (Structural Similarity Index), PSNR (Peak Signal-to-Noise Ratio), and LPIPS (Learned Perceptual Image Patch Similarity).

[0030] An electronic device for performing image processing by combining a frequency domain and a spatial domain according to another aspect of the present invention comprises: at least one processor; and a memory storing instructions that instruct the at least one processor to perform at least one operation.

[0031] The above at least one operation includes: an operation of obtaining a first image by performing a first image processing on an input image; an operation of obtaining a second image by converting the first image into the frequency domain; an operation of obtaining a third image by performing a second image processing on the second image; and an operation of obtaining a spatial domain-based output image by performing an inverse frequency conversion on the third image.

[0032] The above first image processing is performed using a first artificial neural network based on deep learning that has been pre-trained using training data.

[0033] The above second image processing is performed using a deep learning-based second artificial neural network that has been pre-trained using the above training data. Effects of the invention

[0034] By performing image processing by combining the frequency domain and the spatial domain according to the present invention as described above, the quality degradation problem that occurred in conventional single-domain-based image processing methods can be effectively reduced.

[0035] In particular, as spatial domain-based image processing (e.g., super resolution, denoising, reconstruction models, etc.) and frequency-based image processing are performed simultaneously, image quality indicators such as PSNR (Peak Signal-to-Noise Ratio), SSIM (Structural Similarity Index), and LPIPS can be improved compared to a single neural network-based processing method. Brief explanation of the drawing

[0036] FIG. 1 is a conceptual diagram showing the operating environment of an electronic device that performs image processing by combining a frequency domain and a spatial domain according to one embodiment. FIG. 2 is a conceptual diagram illustrating the main operation of an electronic device that performs image processing by combining the frequency domain and the spatial domain according to one embodiment. FIG. 3 is a block diagram showing the functional configuration of an electronic device that performs image processing by combining the frequency domain and the spatial domain according to one embodiment. FIG. 4 is a conceptual diagram for explaining the operation when the image processing units of an electronic device that performs image processing by combining the frequency domain and the spatial domain according to one embodiment are implemented as an artificial neural network. FIG. 5 is a conceptual diagram for explaining the operation of a performance improvement unit according to one embodiment. FIG. 6 is a diagram illustrating the hardware configuration of an electronic device according to one embodiment. Specific details for implementing the invention

[0037] The present invention is susceptible to various modifications and may have various embodiments; specific embodiments are illustrated in the drawings and described in detail in the detailed description. However, this is not intended to limit the invention to specific embodiments, and it should be understood that the invention includes all modifications, equivalents, and substitutions that fall within the spirit and scope of the invention. Similar reference numerals have been used for similar components in the description of each drawing.

[0038] Terms such as first, second, A, B, etc., may be used to describe various components, but said components should not be limited by said terms. These terms are used solely for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the first component may be named the second component, and similarly, the second component may be named the first component. The term "and / or" includes a combination of a plurality of related described items or any of a plurality of related described items.

[0039] When it is stated that one component is "connected" or "connected" to another component, it should be understood that while it may be directly connected or connected to that other component, there may also be other components in between. On the other hand, when it is stated that one component is "directly connected" or "directly connected" to another component, it should be understood that there are no other components in between.

[0040] The terms used in this application are used merely to describe specific embodiments and are not intended to limit the invention. The singular expression includes the plural expression unless the context clearly indicates otherwise. In this application, terms such as "comprising" or "having" are intended to specify the presence of the features, numbers, steps, actions, components, parts, or combinations thereof described in the specification, and should be understood as not precluding the existence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

[0041] Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. Terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and should not be interpreted in an ideal or overly formal sense unless explicitly defined in this application.

[0042] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings.

[0044] FIG. 1 is a conceptual diagram showing the operating environment of an electronic device that performs image processing by combining a frequency domain and a spatial domain according to one embodiment.

[0045] Referring to FIG. 1, an electronic device (100) receives an input image from a user terminal (200), improves the image quality of the input image by combining frequency domain-based image processing and spatial domain-based image processing, and then provides an output image corresponding to the processing result back to the user terminal (200).

[0046] The user terminal (200) is a computing device that transmits input images and receives output images, and optionally, the user terminal (200) may be a device that includes a camera sensor to perform an image capturing function.

[0047] Additionally, the user terminal (200) may load an image file from a storage medium or receive an image file from a network-based shooting system (e.g., CCTV, IoT video sensor, etc.) and then transmit it to the electronic device (100).

[0048] The method of communication between the electronic device (100) and the user terminal (200) may vary depending on the implementation type. For example, they may communicate with each other using wireless communication such as Wi-Fi, Bluetooth, BLE, UWB, NFC, 4G / 5G mobile communication, wireless LAN, ZigBee, etc. In another example, they may communicate with each other using USB, Thunderbolt, Serial (RS-232), Ethernet (LAN), HDMI, MIPI interface, etc. In yet another example, they may communicate with each other using hybrid communication that automatically switches between wireless and wired communication depending on the situation.

[0049] Additionally, the user terminal (200) can communicate in a local network (e.g., LAN or private network) connected to the electronic device (100) or in a remote network environment via the Internet. When remote communication is performed, the user terminal (200) can transmit and receive video data based on HTTP / HTTPS, FTP, MQTT, gRPC, WebRTC, or other communication protocols.

[0050] In one embodiment of the present invention, the electronic device (100) may operate independently without a user terminal (200). In this case, the electronic device (100) may load an input image from an external storage device (e.g., USB storage device, SD card, external SSD), a network storage (NAS), or an image storage module included in the electronic device (100) itself. Additionally, an administrator may select an input image, set image processing conditions, and view and check the quality of an output image through a GUI-based setting panel, a CLI (Command Line Interface), a web-based administrator console, or an API call.

[0051] The electronic device (100) can transmit the image processing result (output image) to a user terminal (200) or store and transmit it to an external server, cloud storage, content management system (CMS), database, or other CCTV control server. At this time, the result may be provided in a static image format such as JPEG, PNG, TIFF, RAW, HEIF, WebP, as well as in the form of metadata including image frame data, compressed bitstream, or comparison data between the original and the processing result.

[0052] Examples of electronic devices (100) or user terminals (200) in one embodiment may include intelligent surveillance cameras (e.g., IP cameras, Smart CCTV), CTV video collection servers, control servers, recording servers (Network Video Recorder: NVR), or video data analysis servers (Video Management System: VMS), edge computing devices, communication-enabled desktop computers, laptop computers, notebooks, smartphones, tablet PCs, mobile phones, smart watches, smart glasses, e-book readers, PMPs (portable multimedia players), portable game consoles, navigation devices, digital cameras, DMB (digital multimedia broadcasting) players, digital audio recorders, digital audio players, digital video recorders, digital video players, PDAs (Personal Digital Assistants), etc.

[0054] FIG. 2 is a conceptual diagram illustrating the main operation of an electronic device that performs image processing by combining the frequency domain and the spatial domain according to one embodiment.

[0055] Referring to FIG. 2, the electronic device (100) performs a series of conversion and processing steps to improve image quality from an input image, and these processing steps can be divided into a frequency domain-based image processing step and a spatial domain-based image processing step.

[0056] First, the electronic device (100) performs a frequency transformation on the input image. For example, the electronic device (100) can transform the input image into a frequency component-based representation using at least one of a Fourier Transform, a Discrete Cosine Transform (DCT), or a Wavelet Transform. The result of this transformation appears in the form of data decomposed into low-frequency and high-frequency components, rather than spatial pixel information of the image. The frequency domain representation can clearly distinguish structural patterns, edge components, texture information, etc., within the image, making it advantageous for selectively enhancing or suppressing specific frequency bands.

[0057] Next, the electronic device (100) performs a first image processing on a first image (IMG1) based on the converted frequency domain. The first image processing includes frequency band-based operations and may include reducing noise components within the image, enhancing high-frequency textures and contours, or removing compression artifacts. For example, the electronic device (100) may perform filtering to reduce residual noise of low-frequency components or improve image clarity by adjusting the degree of attenuation or amplification of high-frequency components. Artifacts such as ringing, halo, and overshoot that may occur when specific components of the image are excessively amplified during this processing step can be minimized according to the processing method of the present invention.

[0058] When the first image processing is completed, the electronic device (100) performs the first image processing and then performs an inverse frequency transform on the second image (IMG2) obtained. Through this, the second image (IMG2) is restored to a third image (IMG3), which is a spatial domain-based pixel structure image, and at this stage, the basic image quality improvement effect compared to the input image is already reflected.

[0059] In the subsequent step, the electronic device (100) performs a second image processing on the third image (IMG3) of the restored spatial domain. The second image processing may include operations based on spatial pixel relationships, such as CNN-based super-resolution, histogram correction, non-linear sharpening, color enhancement, exposure correction, and deblurring. In this process, the electronic device (100) can improve the image quality to a level suitable for the end-use environment by correcting the image's brightness contrast, color reproduction, edge naturalness, and visual balance. The execution or intensity of the second image processing step may be automatically adjusted according to image characteristic analysis criteria or preset conditions.

[0060] Finally, the electronic device (100) can obtain an output image through second image processing of the third image (IMG3) and can store the obtained output image or provide it to an external device.

[0061] Meanwhile, as illustrated in FIG. 2, a first embodiment is defined as a case in which a second image processing is performed after frequency conversion, a first image processing, and an inverse frequency conversion. The first embodiment is described as a representative example, but is not limited thereto. For example, a second embodiment may be defined as a case in which a second image processing is performed first on an input image, and the output is provided as an output image after passing through frequency conversion, a first image processing, and an inverse frequency conversion. According to the second embodiment, depending on the processing order, the second image processing may be referred to as the first image processing, and the first image processing as the second image processing, and the image output through the procedure may also be referred to differently depending on the output order.

[0062] That is, according to the second embodiment, the electronic device (100) may be configured to perform the following operations: obtaining a first image by performing a first image processing on an input image; obtaining a second image by converting the first image into a frequency domain; obtaining a third image by performing a second image processing on the second image; and obtaining a spatial domain-based output image by inversely converting the third image.

[0064] FIG. 3 is a block diagram showing the functional configuration of an electronic device that performs image processing by combining the frequency domain and the spatial domain according to one embodiment.

[0065] Referring to FIG. 3, the electronic device (100) may include a frequency converter (101), a first image processing unit (102), a frequency inverse converter (103), and a second image processing unit (104). This configuration is directly linked to the image processing step flow described in FIG. 2, and each component may be implemented independently or in the form of a combined module.

[0066] First, the frequency conversion unit (101) performs the role of converting the input image into a frequency-based representation. For example, the frequency conversion unit (101) can separate the input image into low-frequency and high-frequency components through a Fourier Transform, Discrete Cosine Transform (DCT), Wavelet Transform, or other spectrum analysis-based algorithms. That is, the frequency conversion unit (101) is a step of separating structural patterns, background brightness, texture, and edge information within the image by domain and converting them into a form suitable for subsequent processing.

[0067] The first image processing unit (102) performs first image processing (i.e., frequency band operation) on the first image (IMG1) obtained by frequency conversion. The first image processing unit (102) can perform frequency filtering to reduce noise components included in the low-frequency region or amplify specific frequency bands in the high-frequency region to enhance the detailed contours and textures of the image. Additionally, the first image processing unit (102) can be configured to minimize image artifacts such as halo, ringing, or overshoot that may generally occur during the high-frequency amplification process.

[0068] The frequency inverse transform unit (103) obtains a third image (IMG3) by restoring the second image (IMG2), obtained through the first image processing in the first image processing unit (102), back into a spatial domain-based pixel structure through frequency inverse transform. Through this, the image is converted back into a form perceptible to humans, and at this point, the image has improved quality compared to the initial input image.

[0069] For example, when the frequency transformation unit (101) transforms an image into the frequency domain using a Discrete Fourier Transform (DFT) or a Fast Fourier Transform (FFT), the frequency inverse transformation unit (103) can perform a restoration operation using an inverse Discrete Fourier Transform (IDFT) or inverse Fast Fourier Transform (IFFT) algorithm paired with the transformation. At this time, the frequency inverse transformation unit (103) receives complex frequency components expressed in real and imaginary parts as input and reconstructs them into pixel values ​​in the time (or space) domain by applying an inverse transformation kernel to each component. If necessary, zero-padding, windowing, and normalization coefficient adjustment may be performed together during the inverse transformation process.

[0070] As another example, when the frequency transformation unit (101) uses an output image of Discrete Cosine Transform (DCT) widely used in image compression standards (JPEG, HEVC, etc.), the inverse frequency transformation unit (103) can apply an output image of Inverse Discrete Cosine Transform (IDCT) to restore the frequency coefficient blocks to the original spatial domain blocks. In this case, the inverse frequency transformation unit (103) can receive frequency coefficients in each block unit (e.g., 8x8, 16x16, etc.), perform IDCT operations, and perform the function of placing the restored pixel blocks back into the entire image coordinate system.

[0071] Additionally, when the frequency conversion unit (101) decomposes the image into multi-resolution frequency bands using a wavelet transform, the frequency inverse transform unit (103) can reconstruct the spatial domain image of the original resolution by integrating a low-frequency subband and one or more high-frequency subbands through the output image inverse wavelet transform output image. At this time, the frequency inverse transform unit (103) can be configured to appropriately combine scale- and direction-specific subbands (horizontal, vertical, diagonal components, etc.) and apply a synthesis filter to hierarchically restore the image.

[0072] In one embodiment, the frequency inverse transform unit (103) may be implemented using only one of the inverse transform algorithms, or it may be implemented to selectively or in combination support IDFT / IFFT, IDCT, inverse wavelet transform, etc., depending on the characteristics of the input image, the purpose of use, or the system settings. Additionally, the frequency transform unit (101) and the frequency inverse transform unit (103) may be designed to share the same transform basis, thereby minimizing information loss during the forward and backward transformation processes and enabling stable image restoration.

[0073] The subsequent second image processing unit (104) performs second image processing on the spatial domain-based third image (IMG3) after frequency inverse transformation is completed. The second image processing unit (104) can perform algorithms such as deep learning-based super-resolution, color correction, sharpness enhancement, gamma adjustment, contrast correction, and non-linear filtering, and thereby can satisfy the overall visual quality of the image and the final quality standards according to the purpose of use.

[0074] Meanwhile, at least one of the first image processing unit (102) and the second image processing unit (104) can be implemented as a deep learning-based artificial neural network.

[0075] For example, the first image processing unit (102) is implemented as a first artificial neural network based on deep learning, and the first artificial neural network may include at least one of a convolutional neural network (CNN) that learns frequency domain characteristics, a wavelet-based network for multiscale frequency processing (Wavelet CNN), a Fourier-based vision model (Fourier Neural Operator, FNO), and a spectral transform-based neural network (Spectral CNN) optimized for frequency domain signal characteristics. Such a model can be used to learn the frequency components of an input image and to remove noise in a specific frequency band, restore patterns, reduce ringing, or enhance high-frequency components.

[0076] Meanwhile, the second image processing unit (104) may be implemented as a second artificial neural network based on deep learning. For example, the second artificial neural network may include at least one of a super-resolution model such as SRCNN, ESRGAN, EDSR, a denoising network based on U-Net structure, a deblurring model based on deep learning, and a vision model based on Transformer (ViT, Swin-Transformer) as a deep learning model suitable for spatial domain-based image improvement. Such a model can perform post-processing functions such as image sharpness improvement, contrast enhancement, color balance correction, or low-light correction by analyzing the structure, contours, texture, and color information of the image based on space.

[0077] The electronic device (100) may further include a performance improvement unit (105). The performance improvement unit (105) may be configured to perform an operation to improve the processing quality of at least one of a first image processing operation performed in a first image processing unit (102) or a second image processing operation performed in a second image processing unit (104). For example, the performance improvement unit (105) may include image quality analysis, model optimization, algorithm parameter tuning, or a learning-based performance enhancement function, and may operate to repeatedly improve the results of the first image processing and / or the second image processing.

[0078] For example, when the first image processing unit (102) and the second image processing unit (104) are implemented as a deep learning-based first artificial neural network and a second artificial neural network, respectively, the performance improvement unit (105) may perform supervised learning or semi-supervised learning using training data to train the first artificial neural network and the second artificial neural network. The training data may consist of training input data and correct answer data. The training input data includes training input images collected in advance for training.

[0079] The ground truth data may include a spatial domain-based ground truth image (GT) and a frequency domain-based frequency ground truth image (GTIMG) obtained by frequency transforming the ground truth image (GT).

[0080] In one embodiment, the training input image may be an image generated by artificially applying various forms of image quality distortion or degradation processing, such as noise (Gaussian noise, salt-and-pepper, Poisson noise, etc.), compression artifact (JPEG compression, block noise), blur (motion blur, Gaussian blur), and illumination imbalance (under / over exposure), to the ground truth image.

[0081] The performance improvement unit (105) receives training data and can perform operations on an objective function, a loss function, and a backpropagation algorithm (the operation of calculating the loss function can be described as calculating the loss below), and can improve the processing performance of the model by repeatedly adjusting network parameters by measuring the difference between the output image of the artificial neural network and the correct data during the training process using the training data.

[0082] In addition, the performance improvement unit (105) may quantitatively evaluate the quality of the image (second image (IMG2) or output image) obtained through the first image processing or second image processing based on image quality indicators such as SSIM (Structural Similarity Index), PSNR (Peak Signal-to-Noise Ratio), and LPIPS (Learned Perceptual Image Patch Similarity), and may use the evaluation results to adjust the weights, activation functions, or hyperparameters of the artificial neural network constituting the first image processing unit (102) or the second image processing unit (104).

[0083] For example, the performance improvement unit (105) compares the output image with the ground truth image (GT) to calculate each image quality index. For example, the image quality index includes SSIM (Structural Similarity Index), PSNR (Peak Signal-to-Noise Ratio), and LPIPS (Learned Perceptual Image Patch Similarity). SSIM (Structural Similarity Index) evaluates image quality based on the structural similarity of two images being compared, and the closer the SSIM value is to 1, the more similar it is to the ground truth image; PSNR (Peak Signal-to-Noise Ratio) measures the error between two images being compared in decibels (dB), and the higher the value, the less distortion there is; and LPIPS (Learned Perceptual Image Patch Similarity) measures perceptual differences similar to human visual characteristics based on a pre-trained neural network (e.g., VGG, AlexNet), and the smaller the value, the more similar the image.

[0084] In an exemplary embodiment, the performance improvement unit (105) may calculate a comprehensive quality score (QScore) by combining at least one or two or more of the image quality indicators, and, for example, the quality score may be defined as in the following mathematical formula 1.

[0085]

[0086] In mathematical formula 1, α, β, and γ are coefficient values ​​that are pre-set according to the usage environment, data characteristics, and purpose criteria (e.g., sharpening enhancement, noise suppression priority, etc.), and SSIM, PSNR, and LPIPS represent each of the image quality indicators described above.

[0087] The performance improvement unit (105) may be configured to perform a parameter update or fine-tuning procedure on the artificial neural network of the first image processing unit (102) or the second image processing unit (104) when the calculated quality score (Qscore) is lower than a preset threshold.

[0088] In another embodiment, the performance improvement unit (105) monitors the rate of change (Q) of the image quality indicator (or quality score) when the image quality indicator is output up to a predetermined n-th output image (n is a natural number greater than or equal to 2) and can perform learning rate decay, early stopping, or model switching when the rate of change falls below a predetermined threshold value.

[0089] The performance improvement unit (105) may be executed in an online learning, transfer learning, self-supervised learning, or reinforcement learning manner depending on user feedback, environmental changes, changes in input video quality, or operating conditions. For example, in a video environment where images are collected for a long time, such as CCTV footage, the performance improvement unit (105) may tune the performance of the artificial neural network in a real-time or post-batch manner to maintain video processing performance optimized for the actual system operating environment, such as specific lighting conditions, sensor characteristics, and changes in compression ratio.

[0091] FIG. 4 is a conceptual diagram for explaining the operation when the image processing units of an electronic device that performs image processing by combining the frequency domain and the spatial domain according to one embodiment are implemented as an artificial neural network.

[0092] Referring to FIG. 4, when an input image is converted into a frequency domain-based first image (IMG1) through a frequency conversion unit (101), a first image processing unit (102) performs first image processing on the first image (IMG1) to generate or output a second image (IMG2), which is a frequency-based improved image. At this time, a performance improvement unit (105) calculates a first loss (Loss1) by comparing the difference between the second image (IMG2) obtained as the output of the first image processing unit (102) and the frequency correct image (GTIMG). The first loss can be defined as at least one of, for example, Mean Squared Error (MSE), L1 loss, Spectrum-based Loss (Frequency Domain Loss), or Structural Similarity-based Loss (SSIM Loss), and is used as a learning criterion for judging the quality of computation performed in the frequency domain by the first artificial neural network of the first image processing unit (102).

[0093] The frequency inverse transform unit (103) converts the second image (IMG2) back into a third image (IMG3) based on the spatial domain, and the second image processing unit (104) performs a spatial-based image quality enhancement operation on the third image (IMG3) to generate or output an output image. At this time, the performance improvement unit (105) calculates a second loss (Loss2) based on the difference between the output image and the correct image (GT). The second loss may consist of, for example, MSE loss, perceptual loss, GAN-based discriminator loss, content loss, or a composite loss based on an image quality measurement index, and this is used as a criterion for evaluating the quality of the operation performed by the second image processing unit (104) during the spatial domain enhancement process.

[0094] In one embodiment, the performance improvement unit (105) can perform supervised learning by adjusting the weights of the first artificial neural network and the second artificial neural network using the first loss and the second loss.

[0095] For example, the performance improvement unit (105) can adjust the weights of the first artificial neural network so that the first loss is minimized, and adjust the weights of the second artificial neural network so that the second loss is minimized.

[0096] In another embodiment, the performance improvement unit (105) may calculate an integrated loss (Ltotal) by combining the first loss and the second loss in a single or weighted sum form. For example, a loss function corresponding to the integrated loss may be defined according to Equation 2 below.

[0097]

[0098] In mathematical formula 2, p and q are predefined values ​​that can be set according to the learning environment, image type, or quality standard, and L1 and L2 may be the first loss and the second loss, respectively. The performance improvement unit (105) can repeatedly update the weights (filter weights, biases, normalization coefficients, etc.) of the first artificial neural network of the first image processing unit (102) and the second artificial neural network of the second image processing unit (104) by performing a backpropagation algorithm to minimize these integrated losses.

[0099] Additionally, the performance improvement unit (105) can perform optimization strategies such as learning rate scheduling, weight reset, regularization, and data augmentation when the quality of the image processing result is below a certain standard, and can also perform mutual correction operations based on feature sharing, residual feedback, and cross-domain attention between the first artificial neural network and the second artificial neural network as needed.

[0100] Accordingly, the performance improvement unit (105) trains the first artificial neural network and the second artificial neural network, either individually or integrally, using the first loss and the second loss, thereby improving the frequency characteristics and spatial visual quality of the image in a balanced manner. Through this, the present invention can provide an image processing system that effectively reduces quality degradation, processing bias, and image distortion problems that may occur in a single domain-based image improvement method, while continuously improving performance even in a long-term operating environment.

[0101] Meanwhile, in the case according to the second embodiment, the input image passes through the second image processing unit (104) to output the first image, the first image passes through the frequency conversion unit (101) to convert into a frequency domain-based second image, and then the second image is input to the first image processing unit (102), and a frequency domain-based third image is output as the output of the first image processing unit (102), and the third image is frequency-inversely converted through the frequency inverse conversion unit (103) to finally output a spatial domain-based output image.

[0102] In the second embodiment, the performance improvement unit (105) can be interpreted as calculating a first loss (Loss1) by comparing the difference between the third image obtained as the output of the first image processing unit (102) and the frequency correct image (GTIMG), and calculating a second loss (Loss2) based on the difference between the first image and the correct image (GT).

[0103] In addition, in the second embodiment, the first to third images are named according to the output order of image processing and should be understood as being different from the first to third images according to the first embodiment described above.

[0104] In addition, considering that the operation of the second image processing unit (104) is performed first in the second embodiment, the image processing process of the second image processing unit (104) may be referred to as the first image processing, and the image processing process of the first image processing unit (102) may be referred to as the second image processing. Also, in the second embodiment, the first artificial neural network of the first image processing unit (102) and the second artificial neural network of the second image processing unit (104) may be referred to as the second artificial neural network and the first artificial neural network, respectively.

[0106] FIG. 5 is a conceptual diagram for explaining the operation of a performance improvement unit according to one embodiment.

[0107] In one embodiment of the present invention, training data may be collected by classifying them according to environmental conditions including lighting environment, shooting conditions, weather conditions, playback conditions, object types, etc. Lighting environment includes indoor / outdoor, natural light, LED, night IR shooting, low-light lighting environment, etc. Shooting conditions include shooting in a stationary state, shooting while moving, shaking, focus errors, etc. Weather conditions include fog, rain, snow, backlight, smoke, etc. Playback conditions include real-time streaming video, compressed video, video with packet loss, etc., and object types include people, vehicles, buildings, indoor structures, document images, natural landscapes, etc.

[0108] In one embodiment, the correct answer data may include a spatial domain-based correct answer image (GT) collected by classifying it according to environmental conditions as described above, and a frequency domain-based frequency correct answer image (GTIMG) obtained by frequency transforming the correct answer image (GT).

[0109] In one embodiment, the performance improvement unit (105) may configure a learning dataset including at least one of a first learning data collected according to a lighting environment, a second learning data collected according to a shooting condition, a third learning data collected according to a weather condition, a fourth learning data collected according to a playback condition, and a fifth learning data collected according to an object type, and may store at least one of a first setting value of hyperparameters of a first artificial neural network and a second artificial neural network learned using the first learning data, a second setting value of hyperparameters of a first artificial neural network and a second artificial neural network learned using the second learning data, a third setting value of hyperparameters of a first artificial neural network and a second artificial neural network learned using the third learning data, a fourth setting value of hyperparameters of a first artificial neural network and a second artificial neural network learned using the fourth learning data, and a fifth setting value of hyperparameters of a first artificial neural network and a second artificial neural network learned using the fifth learning data.

[0110] Here, each of the first to fifth setting values ​​may be a value that can simultaneously tune two artificial neural networks by including both hyperparameters for the first artificial neural network and hyperparameters for the second artificial neural network.

[0111] Meanwhile, although the explanation focused on hyperparameters, it is not limited to this, and the number of layers and filters constituting the artificial neural network may also be included in the setting values.

[0112] The performance improvement unit (105) can perform environment-adaptive tuning by automatically applying the optimal hyperparameter combination for each situation using the previously stored setting values.

[0113] Referring to FIG. 5, the performance improvement unit (105) can determine environmental conditions corresponding to the input image. For example, if the input image is a night CCTV image, the environmental conditions are determined as a lighting environment corresponding to night shooting.

[0114] The performance improvement unit (105) can select a setting value corresponding to the determined environmental conditions and set the hypermeters of the first artificial neural network and the second artificial neural network using the selected setting value. For example, if the input image is a night CCTV image, the performance improvement unit (105) selects a first setting value learned according to the lighting environment corresponding to night shooting.

[0115] According to the above embodiment, the electronic device (100) evaluates image quality standards based on actual operating environment conditions rather than a simple static model, and continuously adjusts the parameters of an artificial neural network that performs image processing based thereon, thereby adaptively responding to any environmental conditions and maintaining stable image quality improvement performance when operating the system for the purpose of long-term image quality improvement.

[0117] FIG. 6 is a diagram illustrating the hardware configuration of an electronic device (100) according to one embodiment.

[0118] Referring to FIG. 6, the electronic device (100) may include at least one processor (110) and a memory (120) that stores instructions that instruct the at least one processor (110) to perform at least one operation.

[0119] The above at least one operation is interpreted to include at least some of the operations of each of the operations or functional parts of the aforementioned electronic device (100), and a specific description is omitted to prevent redundant description.

[0120] Here, at least one processor (110) may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed.

[0121] The memory (120) may be composed of at least one of a volatile storage medium and a non-volatile storage medium. For example, the memory (120) may be one of read-only memory (ROM) and random access memory (RAM).

[0122] The electronic device (100) may further include a storage device (160) for storing temporary data, input data, intermediate processing data, output data, etc. for performing at least one operation. The storage device (160) may be a flash memory, a hard disk drive (HDD), a solid-state drive (SSD), or various memory cards (e.g., a micro SD card).

[0123] Additionally, the electronic device (100) may include a transceiver (130) that performs communication via a wireless network. Additionally, the electronic device (100) may further include an input interface device (140), an output interface device (150), etc. Each component included in the electronic device (100) may be connected by a bus (170) to communicate with one another.

[0125] The methods according to the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the computer-readable medium may be those specifically designed and configured for the present invention, or they may be those known and available to those skilled in the art of computer software.

[0126] Examples of computer-readable media may include hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions may include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc. The aforementioned hardware devices may be configured to operate as at least one software module to perform the operation of the present invention, and vice versa.

[0127] In addition, the above-described method or device may be implemented by combining all or part of its configuration or function, or by implementing it separately.

[0128] Although the present invention has been described above with reference to preferred embodiments, those skilled in the art will understand that various modifications and changes can be made to the invention without departing from the spirit and scope of the invention as described in the following claims.

[0129] delete Explanation of the symbols delete

Claims

Claim 1 An electronic device that performs image processing by combining a frequency domain and a spatial domain, wherein the electronic device comprises: at least one processor; and a memory that stores instructions instructing the at least one processor to perform at least one operation, wherein the at least one operation comprises: an operation of converting an input image into a frequency domain to obtain a first image; an operation of performing a first image processing on the first image to obtain a second image; and an operation of inversely converting the second image to a frequency domain to obtain a third image based on the spatial domain. and an operation of obtaining an output image by performing a second image processing on the third image; wherein the first image processing is performed using a deep learning-based first artificial neural network pre-trained using training data, and the second image processing is performed using a deep learning-based second artificial neural network pre-trained using the training data, and the first artificial neural network includes at least one of a convolutional neural network that has learned frequency domain characteristics, a wavelet-based network for multiscale frequency processing, a Fourier Neural Operator (FNO) vision model, and a Spectral Transform CNN, and the second artificial neural network includes at least one of a super-resolution model, a deep learning-based deblurring model, and a Transformer-based vision model, and the at least one operation includes an operation of calculating an image quality index by comparing the output image and the ground image; and an operation of monitoring the rate of change of the image quality index when output up to a pre-defined n-th output image (where n is a natural number greater than or equal to 2). The electronic device further comprises: an operation to perform at least one of learning rate decay, early stopping, and model switching when the rate of change falls below a predetermined reference threshold. Claim 2 An electronic device according to claim 1, wherein the learning data comprises learning input data and correct answer data, and the correct answer data comprises a pre-collected spatial domain-based correct answer image and a frequency domain-based frequency correct answer image obtained by frequency transforming the correct answer image. Claim 3 An electronic device, wherein, in claim 2, the at least one operation further comprises: an operation of calculating a first loss by comparing the difference between the second image and the frequency correct image; an operation of calculating a second loss based on the difference between the output image and the spatial domain-based correct image; and an operation of adjusting the weights of the first artificial neural network and the second artificial neural network using the first loss and the second loss. Claim 4 An electronic device in claim 3, wherein the operation of adjusting the weights includes adjusting the weights of the first artificial neural network so as to minimize the first loss and adjusting the weights of the second artificial neural network so as to minimize the second loss. Claim 5 In claim 2, the training input image included in the training input data is generated by artificially applying at least one image quality distortion or degradation treatment among noise, compression artifact, blur, and illumination imbalance to the correct answer image. Claim 6 In claim 3, the learning data is collected by classifying according to environmental conditions including at least one of a lighting environment, shooting conditions, weather conditions, playback conditions, and object types; the lighting environment includes at least one of an indoor environment, an outdoor environment, a natural light environment, an LED lighting environment, a night IR shooting environment, and a low-light lighting environment; the shooting conditions include at least one of shooting in a stationary state, shooting while moving, shooting shake, and shooting focus error; the weather conditions include at least one of fog, rain, snow, backlight, and smoke; the playback conditions include at least one of playback of a real-time streaming video, playback of a compressed video, and playback of a video containing packet loss; and the object types include at least one of an object included in the video being a person, a vehicle, a building, an indoor structure, a document image, or a natural landscape. Claim 7 In claim 6, the at least one operation comprises: an operation of configuring a learning dataset including at least one of a first learning data collected according to the lighting environment, a second learning data collected according to the shooting condition, a third learning data collected according to the weather condition, a fourth learning data collected according to the playback condition, and a fifth learning data collected according to the object type; and an operation of storing at least one of a first setting value for the first artificial neural network and the second artificial neural network learned using the first learning data, a second setting value for the first artificial neural network and the second artificial neural network learned using the second learning data, a third setting value for the first artificial neural network and the second artificial neural network learned using the third learning data, a fourth setting value for the first artificial neural network and the second artificial neural network learned using the fourth learning data, and a fifth setting value for the first artificial neural network and the second artificial neural network learned using the fifth learning data; an electronic device. Claim 8 An electronic device, wherein, in claim 7, the at least one operation further comprises: an operation of determining an environmental condition corresponding to the input image; an operation of determining a setting value corresponding to the determined environmental condition; and an operation of setting the first artificial neural network and the second artificial neural network using the determined setting value. Claim 9 An electronic device, wherein, in claim 3, the operation of adjusting the weights comprises: an operation of calculating an integrated loss using the first loss and the second loss; and an operation of adjusting the weights of the first artificial neural network and the second artificial neural network by performing a backpropagation algorithm so as to minimize the integrated loss. Claim 10 An electronic device that performs image processing by combining a frequency domain and a spatial domain, wherein the electronic device comprises: at least one processor; and a memory that stores instructions instructing the at least one processor to perform at least one operation, wherein the at least one operation comprises: an operation of obtaining a first image by performing first image processing on an input image; an operation of obtaining a second image by converting the first image into a frequency domain; and an operation of obtaining a third image by performing second image processing on the second image. and an operation of obtaining a spatial domain-based output image by performing an inverse frequency transform on the third image; wherein the first image processing is performed using a deep learning-based first artificial neural network pre-trained using training data, and the second image processing is performed using a deep learning-based second artificial neural network pre-trained using the training data, and the first artificial neural network includes at least one of a convolutional neural network that has learned frequency domain characteristics, a wavelet-based network for multiscale frequency processing, a Fourier Neural Operator (FNO) vision model, and a Spectral Transform CNN, and the second artificial neural network includes at least one of a super-resolution model, a deep learning-based deblurring model, and a Transformer-based vision model, and the at least one operation includes an operation of calculating an image quality index by comparing the output image and the ground truth image; and an operation of monitoring the rate of change of the image quality index when output up to a pre-defined n-th output image (n is a natural number greater than or equal to 2). The electronic device further comprises: an operation to perform at least one of learning rate decay, early stopping, and model switching when the rate of change falls below a predetermined reference threshold.

Images