Fusion image generating method and apparatus, and storage medium storing instructions for performing fusion image generating method

Abstract

A fusion image generating method using a fusion image generating apparatus, according to an embodiment, comprises the steps of: performing temporal and spatial synchronization of color images and thermal images and preprocessing into preset formats; generating multidimensional color image spectrum features from the preprocessed color images by using a color image encoder in a pre-trained neural network; generating a multidimensional thermal image spectrum feature from the preprocessed thermal images by using a thermal image encoder in the pre-trained neural network; and generating fusion images by fusing the multidimensional color image spectrum features and the multidimensional thermal image spectrum features by using a fusion image decoder in the pre-trained neural network.

Description

A method and device for generating a fusion image, and a storage medium containing a command for performing the method for generating a fusion image

[0001] The present invention relates to a method for generating a fusion image and a device for performing the method.

[0002] Ground-based unmanned equipment, such as autonomous vehicles, and various other robotic systems, rely on perceiving and recognizing their surroundings. Sensing the target environment is a prerequisite for this recognition and recognizing. For this reason, these devices and systems utilize a variety of sensors, including cameras, ultrasonic sensors, and laser rangefinders.

[0003] Among various sensors, camera sensors, which provide information most similar to the human eye and offer a high density of information, are utilized in a majority of equipment and systems. Color cameras are primarily used in these types of cameras, while thermal imaging cameras are also used in some surveillance and military equipment.

[0004] However, while color imaging cameras provide detailed information, they are highly susceptible to factors such as light quantity and direction. Thermal imaging cameras, while less sensitive to the sensing environment, lack detailed information. Consequently, there is a growing demand for fusion imaging, which utilizes both sensors to produce a single, fused image, minimizing the impact of sensing conditions and the environment while providing rich information.

[0005] Before the advent of deep learning technology, the method of generating such fusion images involved directly designing and applying fixed-form filters to extract information to be reflected in the fusion images. However, fixed-form filters were not only difficult to apply to various situations, but also had lower performance in information extraction compared to deep learning technology.

[0006] When applying deep learning technology to create a fusion image, it is possible to create a fusion image that reflects richer information by extracting and combining the features necessary for fusion at a higher level through a neural network.

[0007] However, training a neural network requires a large amount of spatially and temporally aligned color and thermal images. This problem arises because the images in different bandwidths are acquired through physically separate sensors, requiring significant time and effort to align the two images. Furthermore, achieving perfect alignment is extremely difficult compared to the effort required. Therefore, to easily apply deep learning-based fusion image generation techniques utilizing neural networks to real-world systems, it is necessary to minimize the effort required to prepare training data for neural network training.

[0008] According to one embodiment, a method and device for generating a fusion image are provided, which generates a fusion image using a pre-trained neural network after preprocessing a color image and a thermal image into a predetermined format and aligning the same.

[0009] However, the problems to be solved by the present invention are not limited to those mentioned above, and other problems to be solved that are not mentioned can be clearly understood by a person having ordinary skill in the art to which the present invention pertains from the description below.

[0010] A method for generating a fusion image using a fusion image generating device according to a first aspect includes the steps of performing temporal and spatial synchronization on a color image and a thermal image and preprocessing them into a preset format; generating multidimensional color image spectral features from the preprocessed color image using an encoder for a color image within a pre-trained neural network; generating multidimensional thermal image spectral features from the preprocessed thermal image using an encoder for a thermal image within the pre-trained neural network; and generating a fusion image by fusion of the multidimensional color image spectral features and the multidimensional thermal image spectral features using a decoder for a fusion image within the pre-trained neural network.

[0011] A device for generating a fusion image according to a second aspect includes a memory including a command; and a processor for executing the command; wherein the processor, by executing the command, performs preprocessing on a color image and a thermal image in a predetermined format and aligns the color image and the thermal image, generates a multidimensional color image spectral feature from the preprocessed color image using an encoder for a color image in a pre-trained neural network, generates a multidimensional thermal image spectral feature from the preprocessed thermal image using an encoder for a thermal image in the neural network, and generates a fusion image by fusion of the multidimensional color image spectral feature and the multidimensional thermal image spectral feature using a decoder for a fusion image in the neural network.

[0012] According to a third aspect, there is provided a non-transitory computer-readable recording medium storing a computer program, wherein the computer program, when executed by a processor, includes instructions for causing the processor to perform a fusion image generation method, including the steps of: performing temporal and spatial synchronization on a color image and a thermal image, and preprocessing them into a preset format; generating multidimensional color image spectral features from the preprocessed color image using an encoder for a color image within a pre-trained neural network; generating multidimensional thermal image spectral features from the preprocessed thermal image using an encoder for a thermal image within the pre-trained neural network; and generating a fusion image by fusion of the multidimensional color image spectral features and the multidimensional thermal image spectral features using a decoder for a fusion image within the pre-trained neural network.

[0013] According to a fourth aspect, a computer program stored in a computer-readable recording medium, wherein the computer program, when executed by a processor, includes instructions for causing the processor to perform a method for generating a fusion image, the method comprising: performing temporal and spatial synchronization on a color image and a thermal image, and preprocessing them into a preset format; generating multidimensional color image spectral features from the preprocessed color image using an encoder for a color image within a pre-trained neural network; generating multidimensional thermal image spectral features from the preprocessed thermal image using an encoder for a thermal image within the pre-trained neural network; and generating a fusion image by fusion of the multidimensional color image spectral features and the multidimensional thermal image spectral features using a decoder for a fusion image within the pre-trained neural network.

[0014] According to one embodiment, a method and device for generating a fusion image are provided, which aligns a color image and a thermal image and preprocesses them into a preset format, and then generates a fusion image using a pre-trained neural network. For example, image fusion can be generated to minimize environmental influences from image information that is greatly affected by external environmental factors such as the direction and intensity of light, so that it can be used in ground unmanned equipment such as autonomous vehicles and various other robotic systems. When a color image camera and a thermal image camera are simultaneously installed and operated in multiple systems utilizing image information and the generated fusion image is used for sensing the target environment, the performance of the system can be improved in remote control (driving), control (driving) assistance, and fully autonomous (automatic) control (driving), as well as the stability can be increased.

[0015] Furthermore, the present invention eliminates the need for temporal and spatial alignment of input images (color images, thermal images), which was essential for training a neural network to generate fusion images. Training can be performed solely with color and thermal image sets, regardless of alignment. This effectively overcomes the difficulty of separately acquiring or preparing aligned images for training.

[0016] FIG. 1 is an exemplary diagram of a fusion image utilization system to which a fusion image generation device according to one embodiment of the present invention can be applied.

[0017] Figure 2 is a configuration diagram of a fusion image generation device according to one embodiment of the present invention.

[0018] FIG. 3 is a schematic diagram conceptually illustrating the function of a fusion image generation program constituting a fusion image generation device according to one embodiment of the present invention.

[0019] FIG. 4 is an exemplary detailed configuration diagram of a neural network included in a fusion image generation program constituting a fusion image generation device according to one embodiment of the present invention.

[0020] FIG. 5 is a flowchart for explaining a fusion image generation method performed by a fusion image generation device according to one embodiment of the present invention.

[0021] The advantages and features of the present invention, and the methods for achieving them, will become clearer with reference to the embodiments described in detail below together with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and may be implemented in various different forms. These embodiments are provided solely to ensure that the disclosure of the present invention is complete and to fully inform those skilled in the art of the scope of the invention, and the present invention is defined solely by the scope of the claims.

[0022] When describing embodiments of the present invention, detailed descriptions of known functions or configurations will be omitted if they are deemed to unnecessarily obscure the gist of the invention. Furthermore, the terms described below are defined in light of their functions in the embodiments of the present invention and may vary depending on the intent or custom of the user or operator. Therefore, their definitions should be based on the overall content of this specification.

[0023] FIG. 1 is an exemplary diagram of a fusion image utilization system to which a fusion image generation device according to one embodiment of the present invention can be applied.

[0024] A color image captured by a color image camera (10) and a thermal image captured by a thermal image camera (20) are input to a fusion image generation device (100) regardless of temporal and spatial alignment, and the fusion image generation device (100) aligns the input color image and thermal image and preprocesses them into a preset format, and then generates a fusion image using a pre-trained neural network and provides the fusion image to a fusion image learning device (30). The fusion image learning device (30) may be a system that requires sensing of a target environment, such as a ground unmanned equipment such as an autonomous vehicle and various other robot systems, and may utilize the fusion image provided by the fusion image generation device (100) as learning data for sensing the target environment.

[0025] Figure 2 is a configuration diagram of a fusion image generation device (100) according to one embodiment of the present invention.

[0026] A fusion image generation device (100) according to one embodiment includes a memory (110) and a processor (120) in which a fusion image generation program (111) is stored, and may further include an input unit (130) and / or an output unit (140). For example, the fusion image generation device (100) may be implemented by a computing device including a computing operation means such as a microprocessor, an input / output interfacing means, etc.

[0027] The memory (110) of the fusion image generation device (100) can further store various types of information required for executing the fusion image generation program (111), and the fusion image generation program (111) can include commands that can be executed by the processor (120). For example, the fusion image generation program (111) can mean software that includes commands programmed to generate a fusion image using a pre-trained neural network after aligning and preprocessing a color image and a thermal image into a preset format.

[0028] The processor (120) of the fusion image generation device (100) can load and execute a fusion image generation program (111) from the memory (110). The processor (120) can perform a fusion image generation method according to an embodiment of the present invention by executing commands of the fusion image generation program (111).

[0029] FIG. 3 is a schematic diagram conceptually illustrating the function of a fusion image generation program (111) according to one embodiment of the present invention, and FIG. 4 is an exemplary detailed schematic diagram of a neural network included in a fusion image generation program (111) constituting a fusion image generation device (100) according to one embodiment of the present invention.

[0030] Referring to FIGS. 3 and 4, the fusion image generation program (111) may include a preprocessing unit (310), a color image encoder (320), a thermal image encoder (330), and a fusion image decoder (340). The preprocessing unit (310), the color image encoder (320), the thermal image encoder (330), and the fusion image decoder (340) are merely conceptual divisions of the functions of the fusion image generation program (111), and are not limited thereto. According to embodiments, the respective functions of the preprocessing unit (310), the color image encoder (320), the thermal image encoder (330), and the fusion image decoder (340) may be merged / separated, and implemented as a series of commands included in one program.

[0031] The preprocessing unit (310) of the fusion image generation program (111) aligns and preprocesses color images and thermal images into a preset format. For example, the preprocessing unit (310) may convert a color image into YCbCr format and then provide the Y channel so that it can be used as an input for a color image encoder (320). In addition, the preprocessing unit (310) may provide a thermal image as an 8-bit, 1-channel so that it can be used as an input for a thermal image encoder (330).

[0032] The color image encoder (320) of the fusion image generation program (111) can be pre-trained to generate multidimensional color image spectral features from a preprocessed color image. For example, the color image encoder (320) can be implemented with five convolution blocks each composed of two 3x3 convolution filters and two activation layers, and four 2x2 max pooling layers, as illustrated in FIG. 4. Such a color image encoder (320) can be referred to as a spectrum-specific encoder.

[0033] The thermal image encoder (330) of the fusion image generation program (111) can be pre-trained to generate multidimensional thermal image spectral features from preprocessed thermal images. For example, the thermal image encoder (330), like the color image encoder (320), can be implemented with five convolution blocks each composed of two 3x3 convolution filters and two activation layers, and four 2x2 max pooling layers. This thermal image encoder (330), like the color image encoder (320), can be referred to as a spectral feature encoder.

[0034] The decoder (340) for fusion images of the fusion image generation program (111) can be pre-trained to generate a fusion image by fusing multidimensional color image spectral features and multidimensional thermal image spectral features. For example, the decoder (340) for fusion images can be implemented with four convolution blocks each composed of two 3x3 convolution filters and two activation layers, and five 2x2 upconvolution layers, as illustrated in FIG. 4. For example, in order to prevent detailed information from being lost during generation of a fusion image, a skip-connection can be applied between the color image encoder (320) / thermal image encoder (330) and the fusion image decoder (340). Such a decoder (340) for fusion images can be referred to as a spectrum-shared decoder.

[0035] This decoder (340) for fusion images can generate a fusion image by integrating the CbCr color information of a color image and the Y of the 1-channel information of a thermal image when generating a fusion image. In order to generate a fusion image, the decoder (340) for fusion images can concatenate multidimensional color image spectral features and multidimensional thermal image spectral features for each scale on a channel basis, and sort the spectral features for each scale in descending order according to the activation level. For this purpose, the pre-learned loss function of the neural network including the encoder for color images (320), the encoder for thermal images (330), and the decoder for fusion images (340) may include a restoration loss function composed of MSE (mean squared error) loss and SSIM (structural similarity index measure) loss for color images and thermal images, respectively, and a fusion loss function that makes it possible for the sum of the gradients of the color images and the gradients of the fusion images to be identical.

[0036] Referring again to FIG. 2, the input unit (130) can receive various information necessary for the processor (120) to perform a fusion image generation method using the fusion image generation program (111). For example, a color image and a thermal image can be input and provided to the preprocessing unit (310).

[0037] The output unit (140) can output various processing data, including a fusion image generated when the processor (120) performs a fusion image generation method. For example, the output unit (140) can include a data interface that can output various processing data, including a fusion image, to external peripheral devices, a communication module that can transmit various processing data through a communication channel, a playback device that can play back various processing data so that it can be recognized visually or audibly from the outside, etc.

[0038] FIG. 5 is a flowchart for explaining a fusion image generation method performed by a fusion image generation device according to one embodiment of the present invention.

[0039] Hereinafter, a method for generating a fusion image that can be performed by a fusion image generating device according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 5.

[0040] First, the processor (120) of the fusion image generation device (100) can load the fusion image generation program (111) and information necessary for executing the fusion image generation program (111) from the memory (110). Then, the processor (120) can perform the fusion image generation method according to one embodiment by executing the command included in the fusion image generation program (111). Accordingly, the fusion image generation device (100) can be implemented by including a preprocessing unit (310), a color image encoder (320), a thermal image encoder (330), and a fusion image decoder (340). Here, the neural network including the color image encoder (320), the thermal image encoder (330), and the fusion image decoder (340) may be in a state of being pre-learned so as to be able to generate a fusion image from a color image and a thermal image.

[0041] A color image (I_v) and a thermal image (I_1) that are not spatially and temporally aligned are input through the input unit (130), and the input color image (I_v) and thermal image (I_1) are input to a fusion image generation program (111) executed by the processor (120).

[0042] Then, the preprocessing unit (310) of the fusion image generation program (111) aligns the color image (I_v) and the thermal image (I_1) and preprocesses them into a preset format. For example, the preprocessing unit (310) may convert the color image (I_v) into YCbCr format and then provide the Y channel so that it can be used as an input for the color image encoder (320). In addition, the preprocessing unit (310) may provide the thermal image as an 8-bit 1-channel so that it can be used as an input for the thermal image encoder (330) (S510).

[0043] Here, the color image encoder (320) of the fusion image generation program (111) can be pre-trained to generate multidimensional color image spectral features from a pre-processed color image. Thus, the pre-trained color image encoder (320) can generate multidimensional color image spectral features for a color image pre-processed by the preprocessing unit (310) (S520).

[0044] In addition, the thermal image encoder (330) of the fusion image generation program (111) can be pre-trained to generate multidimensional thermal image spectral features from a pre-processed thermal image. Thus, the pre-trained thermal image encoder (330) can generate multidimensional thermal image spectral features for a thermal image pre-processed by the preprocessing unit (310) (S530).

[0045] In addition, the decoder (340) for fusion image generation of the fusion image generation program (111) can be pre-trained to generate a fusion image (I-fused) by fusing multidimensional color image spectral features and multidimensional thermal image spectral features. Accordingly, the pre-trained decoder (340) for fusion image can generate a fusion image (I-fused) by fusing the multidimensional color image spectral features provided from the color image encoder (320) and the multidimensional thermal image spectral features provided from the thermal image encoder (330). This decoder (340) for fusion image can generate a fusion image (I-fused) by integrating the CbCr color information of the color image (I_v) and the Y of the 1-channel information of the thermal image (I_1) (S540) when generating the fusion image.

[0046] Here, the decoder (340) for fusion image can perform the function of a feature line-up module that concatenates multidimensional color image spectral features and multidimensional thermal image spectral features for each scale (s) on a channel basis as shown in mathematical expression 1 to generate a fusion image (I-fused), and sorts the spectral features for each scale in descending order according to the activation level.

[0047]

[0048] This feature line-up module function enables identification of meaningful information (gradient / meaningful information) representing the scene during fusion processing for fusion image generation without separate learning.

[0049] Meanwhile, the learned loss function of the neural network including the encoder for color images (320), the encoder for thermal images (330), and the decoder for fusion images (340) can be configured as in mathematical expression 2.

[0050]

[0051] Here, θ is the parameter of the neural network to be learned.

[0052] α is a parameter set by the user (developer) during the network training process. The optimal value can be empirically determined, and for example, it can be set to 5.

[0053] In mathematical expression 1, the larger α is, the stronger the texture / edge information appears in the fused image. However, as the information becomes stronger, the scene information of the overall image may be distorted.

[0054] Mathematical expression 2 can be composed as mathematical expressions 3 and 4.

[0055]

[0056] Here, β, like α, is a parameter set in advance by the user (developer) during the network training process. The optimal value can be empirically determined, and for example, it can be set to 5.

[0057] In mathematical expression 2, the larger β is, the more structural features can be emphasized in the fused image. However, if the information is excessive, the intensity information in the image may be distorted.

[0058]

[0059] Here, ▽ is a filter operator for extracting gradient information from the image. For example, the Laplacian operator can be applied.

[0060] These mathematical expressions 2 to 4 may imply that the pre-learned loss function of the neural network may include a rasterization loss function composed of MSE loss and SSIM loss for the color image (I_v) and the thermal image (I_1), respectively, and a fusion loss function that makes the sum of the gradients of the color image (I_v) and the gradient of the fused image identical. This is because, although it is difficult to estimate the actual scene information of the fused image when the color image (I_v) and the thermal image (I_1) are not aligned with each other, at least the gradient information should be able to be fused.

[0061] As described so far, according to one embodiment, image fusion can be generated to minimize the environmental impact from image information that is greatly affected by external environmental factors such as the direction and intensity of light, so that it can be used in ground unmanned equipment such as autonomous vehicles and various other robotic systems. When a color image camera and a thermal imaging camera are simultaneously installed and operated in various systems utilizing image information and the generated fusion image is used for sensing the target environment, the performance of the system in remote control (driving), control (driving) assistance, and fully autonomous (automatic) control (driving) can be improved, as well as the stability can be increased.

[0062] Furthermore, the present invention eliminates the need for temporal and spatial alignment of input images (color images, thermal images), which was essential for training a neural network to generate fusion images. Training can be performed solely with color and thermal image sets, regardless of alignment. This effectively overcomes the difficulty of separately acquiring or preparing aligned images for training.

[0063] Meanwhile, a computer program may be implemented to include instructions for causing a processor to perform each step included in the method for generating a fusion image according to the above-described embodiment.

[0064] Additionally, a computer program including commands for causing a processor to perform each step included in the method for generating a fusion image according to the above-described embodiment can be recorded on a computer-readable recording medium.

[0065] The combination of each block of the block diagram and each step of the flowchart attached to the present invention may be performed by computer program instructions. These computer program instructions may be installed in an encoding processor of a general-purpose computer, a special-purpose computer, or other programmable data processing equipment, so that the instructions executed by the encoding processor of the computer or other programmable data processing equipment create a means for performing the functions described in each block of the block diagram or each step of the flowchart. These computer program instructions may also be stored in a computer-available or computer-readable memory that can direct a computer or other programmable data processing equipment to implement the functions in a specific manner, so that the instructions stored in the computer-available or computer-readable memory can also produce an article of manufacture that includes an instruction means for performing the functions described in each block of the block diagram or each step of the flowchart. Since the computer program instructions can also be installed on a computer or other programmable data processing device, a series of operational steps are performed on the computer or other programmable data processing device to create a computer-executable process, and the instructions that cause the computer or other programmable data processing device to perform the steps for executing the functions described in each block of the block diagram and each step of the flowchart can also provide steps for executing the functions described in each block of the block diagram and each step of the flowchart.

[0066] Additionally, each block or step may represent a module, segment, or portion of code that includes one or more executable instructions for performing a specific logical function(s). It should also be noted that in some alternative embodiments, the functions mentioned in the blocks or steps may occur out of order. For example, two blocks or steps depicted in succession may be performed concurrently, or the blocks or steps may sometimes be performed in reverse order, depending on the corresponding functionality.

[0067] The above description is merely an illustrative illustration of the technical idea of ​​the present invention, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential quality of the present invention. Therefore, the embodiments disclosed in the present invention are intended to illustrate rather than limit the technical idea of ​​the present invention, and the scope of the technical idea of ​​the present invention is not limited by these embodiments. The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of the rights of the present invention.

Claims

1. A method for generating a fusion image using a fusion image generating device, A step of performing temporal and spatial synchronization on color images and thermal images and preprocessing them into a preset format; A step of generating multidimensional color image spectral features from the preprocessed color image using an encoder for color images within a pre-trained neural network; A step of generating multidimensional thermal image spectral features from the preprocessed thermal image using a thermal image encoder within the above-mentioned pre-trained neural network; and A step of generating a fusion image by fusing the multidimensional color image spectral features and the multidimensional thermal image spectral features using a decoder for fusion images within the pre-learned neural network; Method for creating a fusion image.

2. In paragraph 1, The above preprocessing step converts the color image into data in YCbCr format, extracts Y channel data from the converted YCbCr format data and provides it as input to the color image encoder, and provides the thermal image composed of single channel data as input to the thermal image encoder. The step of generating the above fusion image is to generate the fusion image by combining the data of the CbCr channel among the data of the YCbCr format of the color image and the data of the single channel of the thermal image as data of the Y channel. Method for creating a fusion image.

3. In paragraph 1, The step of generating the above fusion image is to concatenate multidimensional color image spectral features and multidimensional thermal image spectral features for each scale on a channel basis, and sort the spectral features for each scale in descending order according to the activation level. Method for creating a fusion image.

4. In paragraph 1, The pre-learned loss function of the above neural network includes a restoration loss function composed of MSE (mean squared error) loss and SSIM (structural similarity index measure) loss for the color image and the thermal image, respectively, and a fusion loss function that makes the sum of the gradients of the color image and the gradient of the fused image identical. Method for creating a fusion image.

5. Memory containing instructions; and a processor for executing the above instructions; The processor performs temporal and spatial synchronization for a color image and a thermal image by executing the command, preprocesses the color image in a preset format, generates multidimensional color image spectral features from the preprocessed color image using an encoder for a color image in a pre-trained neural network, generates multidimensional thermal image spectral features from the preprocessed thermal image using an encoder for a thermal image in the pre-trained neural network, and fuses the multidimensional color image spectral features and the multidimensional thermal image spectral features using a decoder for a fusion image in the pre-trained neural network to generate a fusion image. Fusion image generation device.

6. In paragraph 5, The above processor, During the above preprocessing, the color image is converted into data in YCbCr format, and among the converted YCbCr format data, Y channel data is extracted and provided as input to the color image encoder, and the thermal image composed of single channel data is provided as input to the thermal image encoder. During the generation of the above fusion image, among the data of the YCbCr format of the color image, the data of the CbCr channel and the data of the single channel of the thermal image are combined as data of the Y channel to generate the above fusion image. Fusion image generation device.

7. In paragraph 5, The above processor, During the generation of the above fusion image, the multidimensional color image spectral features and the multidimensional thermal image spectral features for each scale are linked on a channel basis, and the spectral features for each scale are sorted in descending order according to the degree of activation. Fusion image generation device.

8. In paragraph 5, The pre-learned loss function of the above neural network includes a rasterization loss function composed of MSE loss and SSIM loss for the color image and the thermal image, respectively, and a fusion loss function that makes the sum of the gradients of the color image and the gradient of the fused image the same. Fusion image generation device.

9. A non-transitory computer-readable recording medium storing a computer program, The above computer program, when executed by a processor, A step of preprocessing color images and thermal images into a pre-set format and alignment; A step of generating multidimensional color image spectral features from the preprocessed color image using an encoder for color images within a pre-trained neural network; A step of generating multidimensional thermal image spectral features from the preprocessed thermal image using a thermal image encoder within the neural network; and A method for generating a fusion image, comprising a step of generating a fusion image by fusing the multidimensional color image spectral features and the multidimensional thermal image spectral features using a decoder for fusion image in the neural network, comprising instructions for causing the processor to perform the method. Non-transitory computer-readable recording medium.

10. In paragraph 9, The above preprocessing step converts the color image into data in YCbCr format, extracts Y channel data from the converted YCbCr format data and provides it as input to the color image encoder, and provides the thermal image composed of single channel data as input to the thermal image encoder. The step of generating the above fusion image is to generate the fusion image by combining the data of the CbCr channel among the data of the YCbCr format of the color image and the data of the single channel of the thermal image as data of the Y channel. Non-transitory computer-readable recording medium.

11. In paragraph 9, The step of generating the above fusion image is to concatenate the multidimensional color image spectral features and multidimensional thermal image spectral features for each scale by channel unit and then sort them in descending order according to the activation level. Non-transitory computer-readable recording medium.

12. In paragraph 9, The pre-learned loss function of the above neural network includes a restoration loss function composed of MSE (mean squared error) loss and SSIM (structural similarity index measure) loss for the color image and the thermal image, respectively, and a fusion loss function that makes the sum of the gradients of the color image and the gradient of the fused image identical. Non-transitory computer-readable recording medium.

Images