Sensor Fusion Image Processing Method and System

Abstract

A sensor fusion image processing method and system are presented. The sensor fusion image processing method proposed in this invention includes the steps of: identifying information and extracting features from input RGB images and infrared images through a feature extraction unit; aligning each feature extracted from the RGB images and infrared images through an image matching unit; aligning the aligned images through an IR image enhancement unit and performing performance enhancement on the aligned infrared images using an adaptive side window technique; and fusing the enhanced infrared images and RGB images through a final image fusion unit.

Description

Technology Field

[0001] The present invention relates to a sensor fusion image processing method and system. Background Technology

[0002] AI technology has made rapid progress in recent years and is achieving remarkable results in various fields. One of the main causes of these achievements is the advancement of computer vision technology for environmental perception. Computer vision plays an essential role in advanced application fields such as autonomous vehicles, robotics, medical image analysis, and smart cities, while specific technologies like semantic segmentation, object detection, and tracking provide key capabilities for AI to understand and interpret its surrounding environment.

[0003] However, conventional single-RGB image-based computer vision technology has a limitation in that it is highly sensitive to lighting conditions. Due to this limitation, performance degradation occurs in low-light environments or environments with glare caused by direct sunlight. To overcome this problem, sensor fusion technology, which combines multi-sensor data including thermal and depth sensors, is gaining increasing attention. By fusing different types of data, sensor fusion can provide rich information that cannot be obtained from a single data source alone.

[0004] RGB images contain high-resolution information such as FullHD (1920x1080) and UHD (3840x2160). However, due to limitations in sensor technology, thermal images have low resolutions such as QQVGA (160x120) and VGA (640x480). A precise alignment process is essential to fuse the two image data sets. However, the difference in resolution between the two sets makes accurate pixel matching difficult, and unexpected noise is generated during the process of upsampling low-resolution images to high resolution, which lowers the reliability of the fusion process. Prior art literature

[0005] [1] Wan, Zifu, et al. "Sigma: Siamese mamba network for multi-modal semantic segmentation." arXiv preprint arXiv:2404.04256 (2024).[2] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku and T. Harada, "MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes," 2017 IEEE / RSJ International Conference on Intelligent Robots and Systems (IROS), Vancouver, BC, Canada, 2017, pp. 5108-5115[3] Yin, Hui, Yuanhao Gong, and Guoping Qiu. "Side window filtering." Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019. 해결하려는 과제

[0006] The technical problem that the present invention aims to solve is to provide a sensor fusion image processing method and system that enables the generation of high-quality data even in low-light or adverse weather environments by combining the respective advantages of RGB images and thermal images, which have complementary characteristics. While RGB images generally possess high resolution, thermal images have relatively low resolution. This difference in resolution makes matching the two images difficult and can cause problems in accurately identifying object locations and detecting boundaries. To solve these problems, the present invention proposes a highly reliable RGB+IR sensor fusion algorithm and improves the resolution of thermal images by applying super-resolution technology. means of solving the problem

[0007] In one aspect, the sensor fusion image processing method proposed in the present invention comprises: a step of identifying information and extracting features from an input RGB image and an infrared image through a feature extraction unit; a step of matching each feature extracted from the RGB image and the infrared image through an image matching unit; a step of aligning the matched images through an IR image enhancement unit and performing performance improvement on the aligned infrared image by utilizing an adaptive side window technique; and a final image fusion step of fusing the improved infrared image and the RGB image through a final image fusion unit.

[0008] The step of identifying information and extracting features from the RGB image and infrared image input through the above feature extraction unit performs a deconvolution operation on the infrared image to improve problems in the image matching process caused by the resolution difference between the RGB image and the infrared image.

[0009] The step of identifying information and extracting features from RGB images and infrared images input through the feature extraction unit above uses a MAP framework for the deconvolution operation and restores the original signal through the probabilistic inference technique of the MAP framework.

[0010] The step of identifying information and extracting features from the RGB image and infrared image input through the feature extraction unit involves extracting a gradient field for the input RGB image and extracting a gradient field for the infrared image improved through deconvolution, and extracting feature points using the ORB (Oriented FAST and Rotated BRIEF) algorithm based on each of the extracted gradient information.

[0011] The step of matching each feature extracted from the RGB image and the infrared image through the image matching unit involves estimating a homography representing the pixel coordinate transformation relationship when registering the infrared image onto the RGB image based on each extracted feature through the RANSAC (RANdom SAmple consensus) algorithm, and matching the RGB image and the infrared image.

[0012] The step of aligning the aligned image through the IR image enhancement unit and performing performance enhancement on the aligned infrared image by utilizing an adaptive side window technique involves detecting an edge direction at a target pixel of the infrared image to perform performance enhancement, and creating a plurality of adaptive side windows based on the detected edge direction.

[0013] The step of aligning the matched image through the IR image enhancement unit and performing performance improvement on the aligned infrared image by utilizing an adaptive side window technique involves calculating multiple candidate values ​​for the generated window using a guide filter that uses an RGB image as a guide image, and selecting the value most similar to the original data among the calculated multiple candidate values ​​as the output value.

[0014] In another aspect, the sensor fusion image processing system proposed in the present invention comprises a feature extraction unit that identifies information and extracts features from input RGB images and infrared images; an image matching unit that matches each feature extracted from the RGB images and infrared images; an IR image enhancement unit that aligns the matched images and performs performance improvement on the aligned infrared images by utilizing an adaptive side window technique; and a final image fusion unit that fuses the enhanced infrared images and RGB images. Effects of the invention

[0015] According to embodiments of the present invention, in order to solve the problem that the accuracy and efficiency may be reduced when fusion data with RGB images due to the low resolution of thermal images and the lack of detailed information caused by this, the resolution of the thermal image is improved through super-resolution technology and detailed information is enhanced to a level similar to that of RGB images, thereby increasing the efficiency and accuracy of data fusion.

[0016] The effects obtainable by improving sensor fusion performance according to an embodiment of the present invention are as follows. First, there is cost reduction. Generally, IR image sensors incur higher costs than RGB image sensors when providing the same resolution. Therefore, super-resolution of thermal images can achieve similar effects without purchasing high-resolution IR sensors, thereby providing cost savings in terms of sensor fusion. This can significantly reduce initial and maintenance costs in cases where sensors need to be installed in large-scale systems or on multiple devices. The second effect is improved stability. The proposed super-resolution technique enhances the overall stability of sensor fusion technology. This enables consistent performance under various environmental conditions, resulting in increased reliability of AI vision systems utilizing it. This improved stability offers significant advantages in various application fields such as autonomous vehicles, security systems, and industrial automation, and strengthens the overall safety and reliability of the system. These effects will improve overall system performance in fields where sensor fusion technology is applied and have a positive impact on various industries. Brief explanation of the drawing

[0017] Figure 1 is a diagram showing an ultra-high resolution process using a dual filter according to the prior art. Figure 2 is a diagram showing an ultra-high resolution processing process using a guide filter according to the prior art. Figure 3 is the architecture of a Sigma image segmentation model according to one embodiment of the present invention. FIG. 4 is a flowchart illustrating a sensor fusion image processing method according to an embodiment of the present invention. FIG. 5 is a diagram illustrating the IR image enhancement process using an adaptive side window technique according to an embodiment of the present invention. FIG. 6 is a drawing for explaining an adaptive side window according to an embodiment of the present invention. FIG. 7 is a diagram showing the configuration of a sensor fusion image processing system according to one embodiment of the present invention. Specific details for implementing the invention

[0018] The performance of single RGB image vision systems often degrades under low-light or adverse weather conditions due to their high dependence on lighting. To address these limitations, RGB+IR sensor fusion technology, which incorporates infrared (IR) images, has emerged as a promising solution. While sensor fusion improves robustness for nighttime and low-light environments, issues arise due to the resolution difference between RGB and IR images. This invention proposes a super-resolution technique for IR images to mitigate these problems. Experiments were conducted using the Sigma model, an image segmentation framework utilizing RGB+IR images. Pre-trained weights were utilized to generate degraded IR images by subsampling the original images to 1 / 8 and 1 / 16 of their resolution. The degraded images were super-resolutioned using bidirectional and guide filter-based methods and used as input for a deep learning-based segmentation model to analyze the model's performance. The results demonstrated that the proposed upsampling method outperforms existing interpolation techniques in segmentation performance. This invention emphasizes that the proposed super-resolution method can reduce problems caused by resolution differences in RGB+IR image sensor fusion. Hereinafter, embodiments of the present invention will be described in detail with reference to the attached drawings.

[0020] RGB images are one of the most widely used formats in digital imaging, representing the color information of objects by combining the three primary colors of light: Red, Green, and Blue channels. Designed to closely resemble human visual perception, RGB images enable vivid colors and detailed representation, making them utilized in various application fields. Due to their high resolution, RGB images offer the advantage of richly expressing object features (e.g., edges, textures, patterns). These characteristics are advantageous for accurate object recognition and detailed analysis, and they are primarily adopted as a fundamental data format in computer vision and image processing systems. However, RGB images have the disadvantage of being heavily dependent on lighting conditions. Under conditions such as low light environments, backlighting, or adverse weather (e.g., rain, fog, snow), image quality degrades, and object identification can become difficult. This dependence on lighting is a major limitation of RGB images, requiring various auxiliary technologies to overcome.

[0021] Thermal imaging is a type of image generated by detecting electromagnetic waves in the infrared region, representing images based on the heat emitted by objects. Thermal imaging is independent of lighting conditions and can detect the heat emission characteristics of objects even in low-light or completely dark environments. Since all objects emit specific infrared radiation depending on their temperature, thermal imaging sensors utilize these characteristics to create images. By utilizing the high penetration power of the Long-Wave Infrared (LWIR) band, image data can be obtained by penetrating visual obstacles such as smoke, fog, and dust. Thermal imaging plays a crucial role in various fields, including night surveillance, rescue operations, medical diagnostics, and industrial monitoring. However, because infrared radiation has a longer wavelength than visible light, sensors that detect it require a larger pixel pitch. Consequently, thermal imaging sensors have the disadvantage of lower resolution than visible light sensors at the same sensor size. Furthermore, infrared sensors are significantly more expensive than RGB sensors due to their complex manufacturing processes and high-cost materials. This acts as a limiting factor in the economic accessibility of thermal imaging systems.

[0022] RGB and thermal images possess complementary characteristics, and fusing them can deliver high performance in various environments. RGB images provide high resolution and rich color information, enabling data representation similar to human visual experience. In contrast, thermal images offer stable results even in low-light conditions due to their illumination independence, heat detection capabilities, and ability to penetrate smoke or fog. Fusing these two image formats combines their respective strengths, making it possible to generate high-quality data even in low-light or adverse weather conditions. However, differences in resolution between the two images can lead to various issues during fusion. While RGB images generally possess high resolution, thermal images have relatively lower resolution. This difference in resolution makes matching the two images difficult and can cause problems in accurately locating objects and detecting boundaries.

[0023] To solve these problems, the present invention proposes a method to improve the resolution of thermal images and minimize the difference from RGB images by applying super-resolution technology to thermal images.

[0024] Recent advancements in various image processing fields, such as object recognition, tracking, image segmentation, and scene recognition, have highlighted the need for high-resolution images to extract more accurate and detailed additional information. High-resolution images play a crucial role in preserving detailed object features and enhancing analysis accuracy. While the most direct method for acquiring high-resolution images is to reduce the pixel size of the image acquisition device, this is constrained by the physical limitations of sensor technology. To address this issue, super-resolution techniques are garnering attention as an important alternative. Super-resolution is a technology that generates high-resolution images by processing low-resolution images; it is noteworthy for its ability to provide high-quality images through software without incurring additional hardware costs. As super-resolution can achieve image quality approaching high resolution while utilizing existing low-resolution sensors, it is becoming increasingly important across a wide range of applications.

[0025] Methods for achieving super-resolution can be broadly classified into statistical, prediction-based, and deep learning-based methods. Statistical methods are simple to implement and require minimal computation, restoring image data through specific mathematical models. Prediction-based methods generate new high-resolution data based on existing data and typically utilize machine learning and optimization techniques. Deep learning-based methods demonstrate powerful performance in restoring or enhancing image details by learning from large amounts of data.

[0026] This invention focuses on a super-resolution technique utilizing statistical methods among various approaches to super-resolution. Statistical methods are useful when introducing super-resolution technology for the first time or in resource-constrained environments because they guarantee a certain level of performance while requiring relatively simple implementation and low computational load. Based on these statistical methods, this invention compares and analyzes the performance between existing methods and the proposed super-resolution technique.

[0028] Figure 1 is a diagram showing an ultra-high resolution process using a dual filter according to the prior art.

[0029] The first super-resolution method is a bilateral filter-based super-resolution method. It is a method that performs super-resolution by utilizing the boundary-preserving properties of a bilateral filter. The super-resolution process using a bilateral filter is shown in Figure 1.

[0030] First, an initial high-resolution image (120) is generated by using an interpolation method (121) on a low-resolution image (110), and super-resolution is performed by applying a bilateral filter (131) (130). The bilateral filter is characterized by reducing blur while better preserving boundaries by considering both spatial adjacency and color value similarity of the image. Natural results can be generated through a non-linear combination using two weights of color and spatial information.

[0032] Figure 2 is a diagram showing an ultra-high resolution processing process using a guide filter according to the prior art.

[0033] The second super-resolution method is the Guided Filter-based super-resolution method. This is a method that effectively restores the structural information of an image. Since the Guided Filter is a linear filter, it requires relatively little computation and is suitable for real-time processing. Because the user can specify a guide input image, RGB images with a lot of detail information can be used as guide inputs to restore detail information more effectively. The super-resolution process using the Guided Filter is shown in Figure 2.

[0034] Similar to the bilateral filter-based super-resolution method, an initial high-resolution image (220) is first generated by using an interpolation method (221) on a low-resolution image (210). The generated initial high-resolution image is used as a basic step to expand the basic shape and size of the low-resolution input image. Subsequently, an RGB image rich in detail information is used as a guide image (230), and super-resolution is performed by applying a Guided Filter (241) (240). The Guided Filter is characterized by a low computational load and an efficient algorithm that enables real-time processing, and is simple yet exhibits powerful performance in maintaining structure and restoring details.

[0035] Guided filters enhance the texture of the input image and sharpen boundaries based on the edge and structural information of the guide image. In this process, they utilize the local linear relationship between the input image and the guide image to enhance details without damaging the edges.

[0037] Figure 3 is the architecture of a Sigma image segmentation model according to one embodiment of the present invention.

[0038] Sigma (Siamese Mamba Network) according to an embodiment of the present invention is an innovative model designed for multi-type image segmentation, and its overall network structure is shown in Fig. 3.

[0039] In the encoding stage (311, 312) of this model, four Visual States Space Blocks (VSSBs) are arranged consecutively, and each VSSB block extracts important features of the image through downsampling. Two encoder branches (311, 312) effectively process different types of image information. Features derived from each branch are combined through a fusion module (320) to form a more sophisticated set of features. The fused features pass to the decoding stage (330), where they undergo upsampling operations and are further enhanced through a Channel-Aware Visual State Block (CAVSB). The CAVSB improves quality by analyzing features more finely on a channel-by-channel basis. Finally, the classifier performs the role of accurately labeling each pixel to a specific class based on the enhanced features.

[0040] The Sigma model according to an embodiment of the present invention was trained using the MFNet dataset. The MFNet dataset is an important resource in the field of multimodal image processing and supports object detection and segmentation under various lighting conditions and environments, including RGB (visible light) and IR (infrared) images. Each pair of images captures the same scene in RGB and thermal images, respectively, enabling research on fusing information between the two spectra. It consists of 2,390 RGB and 2,390 thermal images, with each image having a size of 640x480 pixels and 3 channels. The MFNet dataset includes various environmental conditions, helping researchers simulate diverse experimental conditions such as changes in lighting or weather. Thanks to these characteristics, the MFNet dataset plays a crucial role in the development of new algorithms and the verification of the performance of existing algorithms, and is utilized as an important tool contributing to the advancement of multimodal data fusion technology.

[0041] In this invention, we aim to verify the performance of the proposed super-resolution technique by utilizing a Sigma model. The reconstructed image is used as input data for the Sigma model, and the Sigma model outputs a segmented image based on this input image.

[0042] By analyzing the output results, we evaluate how effectively the super-resolution technique restores the quality of the original image and provides useful information. This performance analysis serves as an important metric for determining the extent to which the proposed super-resolution technique can contribute to improving the resolution of IR images in real-world environments.

[0044] FIG. 4 is a flowchart illustrating a sensor fusion image processing method according to an embodiment of the present invention.

[0045] A sensor fusion performance improvement algorithm according to a sensor fusion image processing method according to one embodiment of the present invention is shown in FIG. 4, and the main process consists of feature extraction (410), image matching (420), IR image improvement (430), and final image fusion (440).

[0046] In step (410), meaningful information is identified and features are extracted from the input sensor data. First, a deconvolution operation is performed on the IR input image (411). IR images may have low resolution and lose detail due to the characteristics of the sensor and the limitations of the lens. Since these characteristics can degrade stability during the image matching process, a MAP (Maximum a posteriori) framework can be used to improve this. FPN (Fixed Pattern Noise) is a spatially fixed noise pattern that occurs in the IR image. This is caused by the physical characteristics of the sensor element or imperfections in the manufacturing process, and refers to noise that is fixed at the same location without changing over time. The mathematical expression of the IR image including FPN is given by Equation (1).

[0047] (1)

[0048] Here, Y is the image measured by the sensor, A is the FPN diagonal matrix, H is the system matrix, X is the actual image, b is the vector noise of the FPN, and n is random noise. The MAP framework is used for deconvolution. MAP is a probabilistic inference technique that reconstructs the most likely original signal based on the observed data. The following equation (2) is the objective function of MAP.

[0049] (2)

[0050] Here, is a normalization parameter. It reflects the strong gradient sparsity of the IR image, so small value, large IR images can be restored by solving the deconvolution problem for the values.

[0051] Next, a gradient field is extracted from the input RGB image (412), and a gradient field is extracted from the IR image improved through deconvolution (413). Based on the respective gradient information, feature points are extracted using the ORB (Oriented FAST and Rotated BRIEF) algorithm (414, 415), and then homography is estimated using the RANSAC (RANdom SAmple consensus) algorithm for image matching.

[0052] The ORB algorithm proceeds in the following order. The first step is corner detection based on the FAST algorithm. The formula is as follows.

[0053] (3)

[0054] Here, R is a value representing corner strength, and pixels with a large R are selected as corners. K is an empirically selected constant (0.04–0.06), and M is a 2x2 matrix generated based on the gradient near the corners of the image.

[0055] (4)

[0056] I x , I y is the gradient in the x and y directions of the image, and pixels with an R value exceeding a threshold are detected as feature points. The direction is calculated from the second detected feature point.

[0057] (5)

[0058] is the direction of the feature point, is the brightness value of the pixel. The corner direction is calculated using formula (5). Then, for rotation invariance Generates a BRIEF descriptor by rotating the sampling pattern according to.

[0059] In step (420), image registration is performed. Homography represents the pixel coordinate transformation relationship when registering an IR image onto an RGB image based on extracted feature points. The formula is as follows.

[0060] (6)

[0061] Here, H is the Homography matrix, (x, y) is the IR pixel coordinate, and (x', y') is the RGB pixel coordinate. Feature point matching may not always be accurate. RANSAC is used to estimate the Homography while excluding outliers. The process of estimating the Homography using RANSAC is as follows.

[0062] First, four pairs of points are randomly selected from the first matched points. Homography is calculated using these four selected points. Second, the error is calculated for the registered point pairs using the calculated Homography. Finally, the above process is repeated to estimate the Homography with the minimum error. Through this, the IR image and the RGB image are accurately aligned. Subsequently, the RGB image is cropped for the Field of View (FoV), and enhancement is performed on the aligned IR image.

[0063] In step (430), the aligned images are aligned (431), and performance enhancement is performed on the aligned IR images (432). The enhancement of the IR images (433) is performed using an adaptive side window technique. At this time, an RGB image rich in detail information is used as a guide image (434). The adaptive side window is described in more detail with reference to FIG. 5.

[0065] FIG. 5 is a diagram illustrating the IR image enhancement process using an adaptive side window technique according to an embodiment of the present invention.

[0066] Referring to FIG. 5, an edge direction is detected at a target pixel (510), and a window is created in eight directions based on the detected direction (520). Multiple candidate values ​​are calculated using the created window (530), and the calculation is performed using a guide filter that uses an RGB image rich in detail information as a guide image. In an embodiment of the present invention, eight candidate values ​​are calculated (531), and among the eight candidate values ​​calculated thereby, the value most similar to the original data is designated as the output value (540).

[0068] FIG. 6 is a drawing for explaining an adaptive side window according to an embodiment of the present invention.

[0069] As shown in FIG. 6, the edge direction at the target pixel is detected (610), and a window is created in eight directions based on the detected direction.

[0070] Eight candidate values ​​are calculated using the generated window, utilizing a guide filter that uses a detail-rich RGB image as a guide image. The formula is as follows.

[0071] (7)

[0072] Here, GF is the guide filter output (candidate value), Each of the 8 windows, is a window parameter, I i is the input image, and the linear coefficients It is calculated as follows.

[0073] (8)

[0074] cov is the covariance between the guide and the input image for each window, is the distribution of guide images for each window, is the local average of the input image for each window, is the local average of the guide images in each window. Among the 8 candidate values ​​calculated from this, the value most similar to the original data is set as the output value.

[0075] (9)

[0076] In this way, noise can be reduced while preserving edge information as much as possible by calculating using a window in the direction where pixels most similar to the original data are located.

[0077] Finally, in step (440), the improved IR image and RGB image are fused to complete the sensor fusion algorithm. This process focuses on overcoming the resolution difference between each sensor and generating a stable integrated image.

[0079] FIG. 7 is a diagram showing the configuration of a sensor fusion image processing system according to one embodiment of the present invention.

[0080] The sensor fusion image processing system (700) according to the present embodiment may include a processor (710), a bus (720), a network interface (730), a memory (740), and a database (750). The memory (740) may include an operating system (741) and a sensor fusion image processing routine (742). The processor (710) may include a feature extraction unit (711), an image matching unit (712), an infrared (IR) image enhancement unit (713), and a final image fusion unit (714). In other embodiments, the sensor fusion image processing system (700) may include more components than those of FIG. 7. However, it is not necessary to clearly illustrate most of the prior art components. For example, the sensor fusion image processing system (700) may include other components such as a display or a transceiver.

[0081] Memory (740) is a computer-readable recording medium and may include a non-perishable permanent mass storage device such as RAM (random access memory), ROM (read only memory), and a disk drive. Additionally, program code for an operating system (741) and a sensor fusion image processing routine (742) may be stored in memory (740). These software components may be loaded from a computer-readable recording medium separate from memory (740) using a drive mechanism (not shown). This separate computer-readable recording medium may include computer-readable recording media (not shown), such as a floppy drive, disk, tape, DVD / CD-ROM drive, or memory card. In another embodiment, software components may be loaded into memory (740) via a network interface (730) rather than a computer-readable recording medium.

[0082] The bus (720) can enable communication and data transmission between components of the sensor fusion image processing system (700). The bus (720) can be configured using a high-speed serial bus, a parallel bus, a Storage Area Network (SAN), and / or other suitable communication technology.

[0083] The network interface (730) may be a computer hardware component for connecting the sensor fusion image processing system (700) to a computer network. The network interface (730) may connect the sensor fusion image processing system (700) to a computer network via a wireless or wired connection.

[0084] The database (750) can serve to store and maintain all information necessary for sensor fusion image processing. Although FIG. 7 illustrates the database (750) being built and included inside the sensor fusion image processing system (700), it is not limited thereto and may be omitted depending on the system implementation method or environment, or it is also possible for all or part of the database to exist as an external database built on a separate system.

[0085] The processor (710) may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input / output operations of the sensor fusion image processing system (700). Instructions may be provided to the processor (710) via memory (740) or a network interface (730) and via a bus (720). The processor (710) may be configured to execute program code for a feature extraction unit (711), an image matching unit (712), an IR image enhancement unit (713), and a final image fusion unit (714). Such program code may be stored in a recording device such as memory (740).

[0086] The feature extraction unit (711), image matching unit (712), IR image enhancement unit (713), and final image fusion unit (714) can be configured to perform the steps (410–440) of FIG. 4.

[0087] The sensor fusion image processing system (700) may include a feature extraction unit (711), an image matching unit (712), an IR image enhancement unit (713), and a final image fusion unit (714).

[0088] A feature extraction unit (711) according to an embodiment of the present invention identifies information and extracts features from an input RGB image and an infrared image.

[0089] The feature extraction unit (711) according to an embodiment of the present invention performs a deconvolution operation on the infrared image to improve the problem in the image matching process caused by the difference in resolution between the RGB image and the infrared image.

[0090] The feature extraction unit (711) according to an embodiment of the present invention uses a MAP framework for the deconvolution operation and restores the original signal through a probabilistic inference technique of the MAP framework.

[0091] A feature extraction unit (711) according to an embodiment of the present invention extracts a gradient field for the input RGB image and extracts a gradient field for the infrared image improved through deconvolution, and extracts feature points using the ORB (Oriented FAST and Rotated BRIEF) algorithm based on each of the extracted gradient information.

[0092] The image matching unit (712) according to an embodiment of the present invention matches each feature extracted from the RGB image and the infrared image.

[0093] The image registration unit (712) according to an embodiment of the present invention estimates a homography representing pixel coordinate transformation relationships when registering an infrared image onto an RGB image based on each of the extracted features through the RANSAC (RANdom SAmple consensus) algorithm, and registers the RGB image and the infrared image.

[0094] The IR image enhancement unit (713) according to an embodiment of the present invention aligns the aligned image and performs performance enhancement on the aligned infrared image by utilizing an adaptive side window technique.

[0095] The IR image enhancement unit (713) according to an embodiment of the present invention detects an edge direction at a target pixel of an infrared image for performing performance enhancement, and creates an adaptive side window in a plurality of directions based on the detected edge direction.

[0096] The IR image enhancement unit (713) according to an embodiment of the present invention calculates a plurality of candidate values ​​using a guide filter that uses an RGB image as a guide image for the generated window, and selects the value most similar to the original data among the calculated plurality of candidate values ​​as the output value.

[0097] The final image fusion unit (714) according to an embodiment of the present invention fuses the improved infrared image and RGB image.

[0099] The device described above may be implemented as a hardware component, a software component, and / or a combination of a hardware component and a software component. For example, the device and components described in the embodiments may be implemented using one or more general-purpose or special-purpose computers, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), a programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing unit may execute an operating system (OS) and one or more software applications executed on said operating system. Additionally, the processing unit may access, store, manipulate, process, and generate data in response to the execution of the software. For ease of understanding, the processing unit may be described as being used as a single unit, but those skilled in the art will understand that the processing unit may include a plurality of processing elements and / or a plurality of types of processing elements. For example, the processing unit may include multiple processors or one processor and one controller. Additionally, other processing configurations, such as parallel processors, are also possible.

[0100] Software may include computer programs, code, instructions, or a combination of one or more of these, and may configure a processing unit to operate as desired or instruct the processing unit independently or collectively. Software and / or data may be embodied in any type of machine, component, physical device, virtual equipment, computer storage medium, or device so as to be interpreted by the processing unit or to provide instructions or data to the processing unit. Software may be distributed over networked computer systems and may be stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

[0101] The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded on a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, etc., either alone or in combination. The program instructions recorded on the medium may be those specifically designed and configured for the embodiment, or they may be those known and available to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical recording media such as CD-ROMs and DVDs; magneto-optical media such as floptical disks; and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, and flash memory. Examples of program instructions include machine code, such as that generated by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

[0102] Although the embodiments have been described above with reference to limited examples and drawings, those skilled in the art can make various modifications and variations from the description above. For example, suitable results can be achieved even if the described techniques are performed in a different order than described, and / or the components of the described system, structure, device, circuit, etc. are combined or assembled in a form different from described, or replaced or substituted by other components or equivalents.

[0103] Therefore, other implementations, other embodiments, and equivalents to the claims also fall within the scope of the claims set forth below.

Claims

Claim 1 A step of identifying information and extracting features from an input RGB image and an infrared image through a feature extraction unit; the step of extracting features comprises generating an improved infrared image by performing a deconvolution operation using a MAP (Maximum a posteriori) framework on the infrared image, extracting a gradient field for the input RGB image and a gradient field for the improved infrared image, respectively, and extracting RGB feature points and infrared feature points, respectively, using an ORB (Oriented FAST and Rotated BRIEF) algorithm based on the extracted gradient information; a step of matching each feature extracted from the RGB image and the infrared image through an image matching unit; the step of matching each feature comprises matching the RGB feature points and the infrared feature points, estimating homography using a RANSAC (RANdom SAmple Consensus) algorithm based on the matched feature points, matching the improved infrared image onto the RGB image using the homography, and matching the RGB image with respect to a field of view A cropping step; a step of aligning the aligned image through an IR image enhancement unit, and performing performance improvement on the aligned infrared image by utilizing an adaptive side window technique;The step of performing the performance improvement comprises: detecting an edge direction at a target pixel of the aligned infrared image; generating adaptive side windows in eight directions based on the detected edge direction; applying a guide filter to each of the adaptive side windows, using the cropped RGB image as a guide image and the aligned infrared image as an input image, to calculate eight candidate values; and selecting the candidate value most similar to the original data among the eight candidate values ​​as the output value of the target pixel to generate an improved infrared image; and a final image fusion step comprising fusing the improved infrared image and the cropped RGB image through a final image fusion unit. Claim 2 In claim 1, the step of identifying information and extracting features from an RGB image and an infrared image input through the feature extraction unit is a sensor fusion image processing method that performs a deconvolution operation on the infrared image to improve problems in the image matching process caused by the resolution difference between the RGB image and the infrared image. Claim 3 In paragraph 2, the step of identifying information and extracting features from RGB images and infrared images input through the feature extraction unit is a sensor fusion image processing method that uses a MAP framework for the deconvolution operation and restores the original signal through a probabilistic inference technique of the MAP framework. Claim 4 In claim 3, the step of identifying information and extracting features from an RGB image and an infrared image input through the feature extraction unit comprises extracting a gradient field for the input RGB image, extracting a gradient field for the infrared image improved through deconvolution, and extracting feature points using an ORB (Oriented FAST and Rotated BRIEF) algorithm based on each of the extracted gradient information. Claim 5 In claim 1, the step of matching each feature extracted from the RGB image and the infrared image through the image matching unit comprises estimating a homography representing a pixel coordinate transformation relationship when registering the infrared image onto the RGB image based on each extracted feature through a RANSAC (RANdom SAmple consensus) algorithm, and matching the RGB image and the infrared image, in a sensor fusion image processing method. Claim 6 In claim 1, the step of aligning the matched image through the IR image enhancement unit and performing performance enhancement using an adaptive side window technique on the aligned infrared image comprises detecting an edge direction at a target pixel of the infrared image for performing performance enhancement and generating an adaptive side window in a plurality of directions based on the detected edge direction. Claim 7 In claim 6, the step of aligning the matched image through the IR image enhancement unit and performing performance improvement on the aligned infrared image by utilizing an adaptive side window technique comprises calculating a plurality of candidate values ​​using a guide filter that uses an RGB image as a guide image for the generated window, and selecting the value most similar to the original data among the calculated plurality of candidate values ​​as an output value. Claim 8 A feature extraction unit that identifies information and extracts features from input RGB images and infrared images; the feature extraction unit generates an improved infrared image by performing a deconvolution operation using a MAP framework on the infrared image, extracts a gradient field for the input RGB image and a gradient field for the improved infrared image, respectively, and extracts RGB feature points and infrared feature points, respectively, using an ORB algorithm based on the extracted gradient information; an image matching unit that matches the features extracted from the RGB images and infrared images; the image matching unit matches the RGB feature points and the infrared feature points, estimates homography through a RANSAC algorithm based on the matched feature points, matches the improved infrared image onto the RGB image using the homography, crops the RGB image for a region of interest; and an IR image enhancement unit that aligns the matched images and performs performance improvement on the aligned infrared images by utilizing an adaptive side window technique. The IR image enhancement unit detects an edge direction at a target pixel of the aligned infrared image, generates adaptive side windows in eight directions based on the detected edge direction, calculates eight candidate values ​​by applying a guide filter that uses the cropped RGB image as a guide image and the aligned infrared image as an input image to each of the adaptive side windows, and generates an improved infrared image by selecting the candidate value most similar to the original data among the eight candidate values ​​as the output value of the target pixel; and the sensor fusion image processing system including a final image fusion unit that fuses the improved infrared image and the cropped RGB image. Claim 9 In claim 8, the feature extraction unit is a sensor fusion image processing system that performs a deconvolution operation on the infrared light to improve problems in the image matching process caused by the resolution difference between the RGB image and the infrared image. Claim 10 In claim 9, the feature extraction unit uses a MAP framework for the deconvolution operation and restores the original signal through a probabilistic inference technique of the MAP framework, thereby forming a sensor fusion image processing system. Claim 11 In claim 10, the feature extraction unit extracts a gradient field for the input RGB image, extracts a gradient field for the infrared image improved through deconvolution, and extracts feature points using the ORB (Oriented FAST and Rotated BRIEF) algorithm based on each of the extracted gradient information, thereby forming a sensor fusion image processing system. Claim 12 In claim 8, the image registration unit estimates a homography representing a pixel coordinate transformation relationship when registering an infrared image onto an RGB image based on each of the extracted features through a RANSAC (RANdom SAmple consensus) algorithm, and is a sensor fusion image processing system that registers the RGB image and the infrared image. Claim 13 In claim 8, the IR image enhancement unit is a sensor fusion image processing system that detects an edge direction at a target pixel of an infrared image for performing performance enhancement and generates an adaptive side window in a plurality of directions based on the detected edge direction. Claim 14 In claim 13, the IR image enhancement unit is a sensor fusion image processing system that calculates a plurality of candidate values ​​using a guide filter that uses an RGB image as a guide image for the generated window, and selects the value most similar to the original data among the calculated plurality of candidate values ​​as an output value. Claim 15 A program stored on a computer-readable storage medium for executing a sensor fusion image processing method, comprising: a step of identifying information and extracting features from an RGB image and an infrared image input through a feature extraction unit; wherein the step of extracting features comprises generating an improved infrared image by performing a deconvolution operation using a MAP (Maximum a posteriori) framework on the infrared image, extracting a gradient field for the input RGB image and a gradient field for the improved infrared image, respectively, and extracting RGB feature points and infrared feature points, respectively, using an ORB (Oriented FAST and Rotated BRIEF) algorithm based on the extracted gradient information; and a step of matching each feature extracted from the RGB image and the infrared image through an image matching unit; wherein the step of matching each feature comprises matching the RGB feature points and the infrared feature points, estimating homography using a RANSAC (RANdom SAmple Consensus) algorithm based on the matched feature points, and using the homography to project the improved infrared image onto the RGB image A step of aligning and cropping the RGB image for the field of view; a step of aligning the aligned image through an IR image enhancement unit and performing performance improvement on the aligned infrared image by utilizing an adaptive side window technique;The step of performing the performance improvement comprises: detecting an edge direction at a target pixel of the aligned infrared image; generating adaptive side windows in eight directions based on the detected edge direction; applying a guide filter to each of the adaptive side windows, using the cropped RGB image as a guide image and the aligned infrared image as an input image, to calculate eight candidate values; and selecting the candidate value most similar to the original data among the eight candidate values ​​as the output value of the target pixel to generate an improved infrared image; and a final image fusion step of fusing the improved infrared image and the cropped RGB image through a final image fusion unit, wherein the program is stored on a computer-readable storage medium.

Images