Image processing device, image processing method and program

Abstract

This allows for improved quality of colored images generated from thermal infrared images. [Solution] An input unit (11) inputs a thermal infrared image of an object captured using a thermal infrared camera. An image generation unit (12) generates a colored image from the thermal infrared image and generates a map representing the object contained in the thermal infrared image from the thermal infrared image. A loss calculation unit (14) calculates a first loss by identifying the colored image based on a visible light image of the object captured using a visible light camera, and calculates a second loss by using the map to identify a second mask image obtained by performing mask processing on the colored image based on a first mask image obtained by performing mask processing on the visible light image. The image generation unit (12) includes a first learning model trained to output a colored image by inputting a thermal infrared image. Parameters of the first learning model are updated using the first and second losses.

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Thermal infrared (TIR) ​​cameras are cameras that capture thermal infrared rays emitted according to the temperature of an object. They are used in a variety of fields because they can capture clear images even in bad weather such as rain, fog, and sandstorms, or in low-light environments such as at night or inside tunnels.

[0003] For example, thermal infrared cameras have been adopted as sensors for taxis in the field of autonomous driving. Furthermore, in the aviation field, systems are being used that superimpose images captured by thermal infrared cameras onto the driver's field of view to assist situational awareness during operation. Furthermore, in the field of maritime rescue, which must be able to adapt to a variety of situations, thermal infrared cameras are being used to search for victims in need of rescue.

[0004] However, images captured using a thermal infrared camera (hereinafter referred to as thermal infrared images) do not contain as much color information as visible light images, making it more difficult to interpret the objects contained in thermal infrared images and the conditions surrounding those objects compared to visible light images.

[0005] For this reason, it has been considered to generate a colored image by coloring the thermal infrared image from the thermal infrared image, but the quality of the colored image is often low. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Laid-Open No. 2024-006730 [Patent Document 2] Special Publication No. 2022-547769 [Patent Document 3] Japanese Patent Publication No. 2022-038287 Summary of the Invention [Problem to be solved by the invention]

[0007] SUMMARY OF THE INVENTION It is therefore an object of the present invention to provide an image processing device, an image processing method, and a program that can improve the quality of a colored image generated from a thermal infrared image. [Means for solving the problem]

[0008] According to one aspect of the present invention, there is provided an image processing device comprising: an input means for inputting a thermal infrared image of an object captured using a thermal infrared camera; a first generation means for generating a colored image by coloring the thermal infrared image from the input thermal infrared image; a second generation means for generating a map representing the object contained in the thermal infrared image from the input thermal infrared image; a first calculation means for calculating a first loss by discriminating the generated colored image based on a visible light image of the object captured using a visible light camera; and a second calculation means for calculating a second loss by discriminating a second mask image obtained by performing mask processing on the colored image based on a first mask image obtained by performing mask processing on the visible light image using the generated map, wherein the first generation means includes a first learning model trained to output the colored image by inputting the thermal infrared image, and parameters of the first learning model are updated using the calculated first and second losses. [Effects of the Invention]

[0009] The present invention makes it possible to improve the quality of colored images produced from thermal infrared images. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing apparatus according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing apparatus. [Figure 3] 10 is a flowchart showing an example of a processing procedure of the image processing apparatus. [Figure 4] FIG. 2 is a diagram specifically illustrating the operation of the image processing device. [Figure 5] FIG. 10 is a diagram showing the relationship between a segmentation module and a coloring module. [Figure 6] FIG. 10 is a diagram for explaining a modified example of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a block diagram showing an example of the functional configuration of an image processing device according to this embodiment. The image processing device 10 shown in Fig. 1 is an information processing device used to generate a colored image (RGB colored image) by coloring a thermal infrared image captured using a thermal infrared sensor.

[0012] As shown in FIG. 1, the image processing device 10 includes an input unit 11, an image generation unit 12, a mask generation unit 13, a loss calculation unit 14, and a learning unit 15.

[0013] A thermal infrared image captured using a thermal infrared sensor is input to the input unit 11. The thermal infrared image input by the input unit 11 includes, for example, at least one object occupying a predetermined area in the thermal infrared image.

[0014] The image generation unit 12 corresponds to a generator that generates a colored image, and includes a coloring module and a segmentation module.

[0015] The coloring module is a module that generates a colored image from a thermal infrared image input by the input unit 11 (converts a thermal infrared image into a colored image). The coloring module includes a network model (first learning model) with an encoder-decoder structure. Hereinafter, the network model included in the coloring module will be referred to as the coloring model.

[0016] The segmentation module is a module that generates a map representing objects included in a thermal infrared image from the thermal infrared image input by the input unit 11. The segmentation module includes a trained semantic segmentation model (second training model).

[0017] Semantic segmentation refers to the task of assigning predefined labels to each pixel in a given image, allowing for pixel-by-pixel classification and abstraction of objects and scenes contained in the image.

[0018] In this embodiment, the segmentation module uses the above-mentioned semantic segmentation model to generate a semantic label map in which each region of the thermal infrared image is associated with a semantic label (class label) that represents the type or meaning of the object contained in that region.

[0019] The mask generation unit 13 generates a mask (hereinafter referred to as a class mask) based on the semantic label map generated by the image generation unit 12. The class mask generated by the mask generation unit 13 is used, for example, to mask areas other than those associated with a predetermined semantic label (i.e., to mask areas other than a predetermined object).

[0020] The loss calculation unit 14 calculates the loss by discriminating the colored image generated by the image generation unit 12 based on an image captured using a visible light camera (hereinafter referred to as a visible light image).

[0021] Specifically, the loss calculation unit 14 includes a full image discrimination model (first discrimination model) that has been trained to discriminate the authenticity of a colored image when compared with a visible light image, and calculates an adversarial loss (first adversarial loss) from the output of the full image discrimination model to which a visible light image and a thermal infrared image are input.

[0022] Furthermore, the loss calculation unit 14 generates a visible light mask image (first mask image) and a colored mask image (second mask image) by applying class masks to the visible light image and the colored image. The visible light mask image corresponds to an image obtained by performing mask processing using the class mask on the visible light image. The colored mask image corresponds to an image obtained by performing mask processing using the class mask on the colored image. In other words, the visible light mask image and the colored mask image correspond to images (specific class images) of only areas to which specific semantic labels (class labels) are assigned.

[0023] The loss calculation unit 14 calculates the loss by determining the colored mask image based on the visible light mask image.

[0024] Specifically, the loss calculation unit 14 includes a class image discrimination model (second discrimination model) that has been trained to discriminate the authenticity of a colored mask image when compared with a visible light mask image, and calculates a class adversarial loss (second adversarial loss) from the output of the class image discrimination model to which the visible light mask image and the colored mask image are input. The class adversarial loss is an adversarial loss calculated from a specific class image (an image of an area not masked by masking).

[0025] As mentioned above, the thermal infrared image is an image captured using a thermal infrared camera, but the visible light image is an image (ground-truth) captured using a visible light camera instead of the thermal infrared camera, and contains the same objects as the thermal infrared image. In other words, the visible light image corresponds to the ground truth data for the colored image.

[0026] The learning unit 15 updates the parameters of the coloring model described above (that is, learns the coloring model) using the losses (the adversarial loss and the class adversarial loss) calculated by the loss calculation unit 14.

[0027] Fig. 2 shows an example of the hardware configuration of the image processing device 10 shown in Fig. 1. As shown in Fig. 2, the image processing device 10 includes a CPU 101, a nonvolatile memory 102, a main memory 103, a communication device 104, and the like.

[0028] The CPU 101 is a processor for controlling the operation of each component in the image processing device 10. The CPU 101 may be a single processor or may be configured with multiple processors. The CPU 101 executes various programs loaded from the nonvolatile memory 102 to the main memory 103. The communication device 104 is a device configured to perform wired or wireless communication with an external device.

[0029] 1 are realized by, for example, the CPU 101 (i.e., the computer of the image processing device 10) executing a predetermined program, that is, by software. This program may be stored in a computer-readable storage medium and distributed, or may be downloaded to the image processing device 10 via a network.

[0030] Here, it has been described that some or all of the units 11 to 15 are executed by software, but some or all of the units 11 to 15 may be realized by hardware such as an IC (Integrated Circuit), or may be realized by a combination of software and hardware.

[0031] 2 shows only the nonvolatile memory 102 and the main memory 103, the image processing device 10 may also include other storage devices such as a hard disk drive (HDD) and a solid state drive (SSD). Although omitted in FIG. 2, the image processing device 10 may also include an input device such as a mouse or a keyboard and a display device such as a display.

[0032] An example of the processing procedure of the image processing device 10 according to this embodiment will be described below with reference to the flowchart in Fig. 3. Note that the image processing device 10 according to this embodiment generates a colored image from a thermal infrared image using the above-described coloring model, and the processing shown in Fig. 3 corresponds to the processing when learning the coloring model.

[0033] First, the input unit 11 inputs a thermal infrared image of an object captured using a thermal infrared sensor (step S1). The thermal infrared image input in step S1 is passed from the input unit 11 to each of the coloring module and segmentation module of the image generation unit 12.

[0034] The segmentation module includes a fully trained semantic segmentation model, which is constructed so as to input a thermal infrared image passed from the input unit 11 and output a semantic label representing the type of object contained in the thermal infrared image on a pixel-by-pixel basis. The segmentation module generates a semantic label map by assigning the semantic label thus output from the semantic segmentation model to each pixel of the thermal infrared image (step S2).

[0035] The coloring module also includes a coloring model that has been trained to input a thermal infrared image passed from the input unit 11 and output a colored image converted from the thermal infrared image, and generates a colored image from the thermal infrared image using the coloring model (step S3).

[0036] Here, the semantic segmentation model is constructed to output the above-mentioned semantic labels, and it is considered that in the process leading up to outputting the semantic labels, the semantic segmentation model extracts features of the thermal infrared image suitable for associating each region of the thermal infrared image with the semantic label. By providing such features of the thermal infrared image to the coloring model as effective information for identifying the meaning (class) of each region of the thermal infrared image, it is expected that each region of the thermal infrared image can be drawn (colored) in an appropriate color based on the meaning.

[0037] That is, in this embodiment, when the semantic label map is generated in step S2 (i.e., when the semantic segmentation model outputs semantic labels), the features of the thermal infrared image extracted by the semantic segmentation model are further input into the coloring model.

[0038] The coloring module may generate the colored image taking into account the semantic label map generated in step S2.

[0039] After the process of step S3 is executed, the loss calculation unit 14 executes a first discrimination process to discriminate the authenticity of the entire image by applying the visible light image captured by the visible light camera and the colored image generated in step S3 to the whole image discrimination model (step S4). The whole image discrimination model is a network model that receives the visible light image and the colored image, and by executing the process of step S4, the adversarial loss is calculated from the output of the whole image discrimination model that has input the visible light image and the colored image.

[0040] Furthermore, the mask generation unit 13 can generate a class mask from the semantic label map generated in step S2. The loss calculation unit 14 executes a second discrimination process to discriminate the authenticity of the mask image (class image) by applying the visible light mask image and the colored mask image generated by executing the mask process using the class mask to the class image discrimination model described above (step S5). The class image discrimination model is a network model that receives the visible light class image and the colored mask image, and by executing the process of step S5, a class adversarial loss is calculated from the output of the class image discrimination model to which the visible light mask image and the colored mask image have been input.

[0041] Next, the learning unit 15 performs learning of the coloring model included in the coloring module using the adversarial loss calculated by executing the process of step S4 (first discrimination process) and the class adversarial loss calculated by executing the process of step S5 (second discrimination process) (step S6).

[0042] In this embodiment, for example, two network models, a coloring model and a full image discrimination model, constitute a generative adversarial network (GAN). According to the process shown in Fig. 3, the coloring model and the full image discrimination model constituting such a generative adversarial network compete with each other, and the parameters (weights) of the coloring model are updated so that a colored image generated by the coloring model is similar to the visible light image to such an extent that the full image discrimination model can discriminate the colored image as a genuine visible light image.

[0043] Furthermore, according to the process shown in Figure 3, by using a class image discrimination model (i.e., class adversarial loss) in addition to the whole image discrimination model described above, it is possible to realize learning of a coloring model that improves local quality.

[0044] Hereinafter, the operation of the image processing device 10 according to this embodiment will be specifically described with reference to FIG.

[0045] As shown in FIG. 4, the image generator 12 (generator) includes a segmentation module 12a and a coloring module 12b, to which the thermal infrared image is input.

[0046] Here, FIG. 5 shows the relationship between the segmentation module 12a and the coloring module 12b in this embodiment.

[0047] The segmentation module 12a includes a semantic segmentation model for thermal infrared images, whose parameters (weights) have been fixed through prior learning. The semantic segmentation model in this embodiment is implemented, for example, by a convolutional neural network (CNN) for segmentation having an encoder-decoder structure. Such a semantic segmentation model can achieve high-speed and highly accurate segmentation by applying convolution that enables depthwise separation.

[0048] The encoder unit at the front end of the semantic segmentation model includes multiple layers that perform processes such as convolution, and the decoder unit at the rear end of the semantic segmentation model includes multiple layers that perform processes such as spatial pyramid pooling, convolution, upsampling, and concatenation. Such a semantic segmentation model can input a thermal infrared image (input image) and output semantic labels that indicate the type of object contained in each region of the thermal infrared image.

[0049] On the other hand, the coloring module 12b includes a coloring model for generating a colored image from a thermal infrared image (input image). In this embodiment, a convolutional neural network with an encoder-decoder structure is used as the coloring model. In other words, the coloring model is a network model constructed so that the encoder acquires the features of the thermal infrared image and then the decoder reconstructs a colored image (RGB image).

[0050] The encoder section before the coloring model includes multiple layers that perform operations such as convolution and max pooling, while the decoder section after the coloring model includes multiple layers that perform operations such as transposed convolution (deconvolution).

[0051] Due to cost and technical issues, edges and textures in thermal infrared images tend to be blurred compared to general visible light images. Therefore, retaining low-level features (information) representing edges and textures is an important factor in the process of coloring thermal infrared images. For this reason, the coloring model in this embodiment employs a U-Net type skip connection that connects corresponding layers of the encoder and decoder, as shown in FIG. 5. By linking the feature map of the encoder to the feature map of the decoder, low-level features extracted in shallower layers are retained. In other words, this embodiment assumes the use of a coloring model that is a U-Net neural network.

[0052] Here, the encoder of the semantic segmentation model functions as a feature extractor that extracts features from the thermal infrared image to output semantic labels, and each layer of the encoder is connected to the coloring model (encoder) by a branch. This structure allows the features of the thermal infrared image extracted by the encoder of the semantic segmentation model to be transmitted to the coloring model.

[0053] The encoder unit of the coloring model receives the features of the thermal infrared image transmitted from the semantic segmentation model, adjusts the resolution and the number of channels by convolution and resizing the features, and combines them with the feature maps (i.e., the features on the coloring module side) output from the corresponding layers of the encoder unit of the coloring model. In other words, some layers (convolutional layers) of the encoder unit of the coloring model operate to receive as input the output of the previous layer and the features of the thermal infrared image transmitted from the semantic segmentation model combined in the channel direction.

[0054] As described above, a semantic segmentation model that has undergone sufficient training is believed to be able to extract features that are effective in matching thermal infrared images with semantic labels (class labels) that represent, for example, cars or people. Therefore, by inputting the features extracted by the semantic segmentation model into a coloring model, the coloring model can be assisted in distinguishing between each region of the thermal infrared image, thereby realizing appropriate color matching (i.e., coloring) for each region.

[0055] Furthermore, the decoder unit of the semantic segmentation model outputs semantic labels, which are also transmitted to the coloring model. In this case, the coloring model includes a connection layer that combines the transmitted semantic labels. Convolution and max-pooling processes are performed at each layer of the encoder unit of the coloring model, resulting in a feature map output from the encoder unit with a lower resolution than the thermal infrared image. The feature map output from the encoder unit is restored to the same resolution as the thermal infrared image (i.e., the input image) through the decoder unit, combined with the semantic label (map) transmitted from the semantic segmentation model, and then fused by convolution. The resulting image is output as a final colored image (RGB image) through an output layer that includes an activation function such as tanh.

[0056] As described above, by inputting semantic labels to the layer (convolutional layer) immediately before the output of the coloring model, semantic information can be given to each pixel in the final process of generating the colored image, and corrections can be made to create consistency in the coloring of each region of the colored image (output image).

[0057] Each layer of the encoder and decoder of the coloring model may incorporate an attention mechanism that promotes attention to important areas of the coloring model while retaining position information of the feature map.

[0058] According to the above-mentioned segmentation module 12a (semantic segmentation model) and coloring module 12b (coloring model), the image generation unit 12 (generator) in this embodiment can receive a grayscale thermal infrared image as input and generate a semantic label map in which a semantic label is assigned to each pixel of the thermal infrared image and an RGB colored image.

[0059] Note that the relationship between the segmentation module 12a and the coloring module 12b described in Figure 5 is an example, and the segmentation module 12a and the coloring module 12b may be configured to operate in conjunction with each other to generate a colored image.

[0060] Returning to FIG. 4 again, the semantic label map generated by the image generation unit 12 is used to calculate the class adversarial loss. In this case, the mask generation unit 13 generates a class mask from the semantic label map. The class mask is used to realize image processing that displays only areas assigned the same semantic label (i.e., areas including a predetermined object) and hides other areas. Note that, for example, if a thermal infrared image includes multiple objects, the class mask may be generated by randomly selecting at least one object (i.e., one or more objects) from the multiple objects, but the one object may also be selected by, for example, a user using the image processing device 10.

[0061] As shown in FIG. 4, the loss calculation unit 14 includes an all-image classifier 14a and a class image classifier 14b.

[0062] The whole image classifier 14a includes a whole image discrimination model, which is a network model that receives the entire colored image generated by the image generation unit 12 (coloring module 12b) and the entire visible light image (Ground-Truth), and operates to discriminate the authenticity of the entire image. The loss calculation unit 14 calculates the adversarial loss from the output of the whole image classifier 14a (whole image discrimination model).

[0063] In this embodiment, a technique called RSGAN (Relativistic Standard GAN) is used as a method for calculating the adversarial loss. In this case, the adversarial loss L adv and the adversarial loss L used to train the full image classification model (full image classifier 14a). D is expressed as the following equations (1) and (2).

number

[0064] In the above equations (1) and (2), x is the input image (thermal infrared image), y is the visible light image (Ground-Truth), G(·) is the output of the generator (coloring model), and D(·) is the output of the whole image discriminator 14a (whole image discrimination model).

[0065] Hostile Losses L adv becomes smaller when the full image classifier 14a judges that the colored image is more likely to be real than the visible light image, and such an adversarial loss L adv According to the method, a coloring model can be trained to facilitate the generation of realistic-looking colored images.

[0066] On the other hand, the adversarial loss L D becomes smaller when the whole image classifier 14a judges that the visible light image is more likely to be real than the colored image, and such an adversarial loss L D According to this, it is possible to learn an all-image discrimination model that promotes correct authentication.

[0067] The class image discriminator 14b includes a class image discrimination model, which is a network model that receives a colored mask image and a visible light mask image generated by the element product of class masks for each of the colored image and the visible light image, and operates to discriminate the authenticity of a specific class image (mask image). The loss calculation unit 14 calculates a class adversarial loss from the output of the class image discriminator 14b (class image discrimination model).

[0068] Here, it is assumed that the above-mentioned semantic label map is composed of H (height) × W (width) pixels, and a semantic label is assigned to each of the pixels. In this case, if the index of the pixel in the width direction (x direction) is i, the index of the pixel in the height direction (y direction) is j, and the randomly selected semantic label (class label) is s, the class mask M is expressed as in the following equation (3).

number

[0069] According to equation (3), the class mask M i,j is a pixel i,j 1 if the pixel (including the area) is not masked i,j It is 0 if (the area including) is masked.

[0070] Furthermore, the class adversarial loss L used to train the coloring model (generator) class and the class adversarial loss L used to train the class image classification model (class image classifier 14b). Dclass are expressed as the following equations (4) and (5).

number

[0071] In the above formulas (4) and (5), D class (·) is the output of the class image discriminator 14b (class image discrimination model).

[0072] Although we will not go into detail here, the class adversarial loss L class and L Dclass is the same as the adversarial loss L described above, except that it is calculated only for the region restricted by the masking process (i.e., for the image after the masking process has been performed). adv and L D is the same as:

[0073] Class adversarial loss L class becomes smaller when the class image discriminator 14b judges that the colored mask image is more likely to be real than the visible light mask image, and such a class adversarial loss L class According to the method, a coloring model can be trained to facilitate the generation of realistic-looking colored mask images.

[0074] On the other hand, the class adversarial loss L Dclass becomes smaller when the class image discriminator 14b judges that the visible light mask image is more likely to be real than the colored mask image, and such a class adversarial loss L DclassAccording to this method, it is possible to learn a class image discrimination model that promotes correct authentication.

[0075] Note that a generative adversarial network called PatchGAN may be adopted for the structures of the all image classifier 14a and the class image classifier 14b.

[0076] Here, in this embodiment, the adversarial loss L adv and L D and the class adversarial loss L class and L Dclass However, it is considered that the quality of the colored image can be further improved by performing learning using an objective function that uses a composite loss (loss function).

[0077] Specifically, in this embodiment, the content loss L con can be added to the objective function. con is the loss imposed to capture the overall color and texture trends of the colored image (generated image), and is expressed as the following equation (6).

number

[0078] In the above (6), ||·||1 is the L1 norm, W is the number of pixels corresponding to the width of the image (colored image or visible light image), and H is the number of pixels corresponding to the height of the image.

[0079] In addition, the content loss L con However, it is difficult to perform learning that reproduces high-level features of visible light images (ground-truth). For this reason, in this embodiment, for example, perceptual loss L per can be further added to the objective function. peris a loss based on high-level features extracted using, for example, a trained VGG-16 (a convolutional neural network with a depth of 16 layers), and is expressed as the following equation (7).

number

[0080] In the above equation (7), φ k (·) is the feature map (high-level feature) output from the kth layer of the trained VGG-16, C k ,W k ,H k are the number of channels, width, and height of the feature map.

[0081] Such perceptual loss L per According to the method, it is possible to train a coloration model that promotes the reproduction of the perceptual features of the visible light image in a colored image.

[0082] Furthermore, the total variation loss L calculated from the gradient in the x and y directions of the colored image, for example, tv can be added to the objective function. Total Variation loss L tv is expressed as the following equation (8).

number

[0083] In the above equation (8), ▽x is the gradient in the x direction, ▽y is the gradient in the y direction, and |·| is the absolute value for each element.

[0084] Such a total variation loss L tv According to the method, it is possible to train a coloring model that promotes reducing noise in colored images and enhancing spatial smoothness.

[0085] Taking into account the above adversarial loss, class adversarial loss, content loss, perceptual loss, and total variation loss, the final objective function L for training the coloring model (generator) is all is expressed as the following equation (9).

number

[0086] In addition, λ in Equation (9) adv , λ class , λ con , λ per and λ tv is the weight for each loss (loss function), e.g., λ adv =λ class = 0.03, λ con =λ per =λ tv =1.

[0087] Here, we assume that training is performed using all of the adversarial loss, class adversarial loss, content loss, perceptual loss, and total variation loss. However, at least one of these losses may be omitted, or losses other than these losses may also be used.

[0088] As described above, the image processing device 10 according to this embodiment receives a thermal infrared image of an object captured using a thermal infrared camera, generates a colored image by coloring the input thermal infrared image, and generates a map representing the object included in the thermal infrared image from the input thermal infrared image. The image processing device 10 according to this embodiment also calculates a first loss by identifying a colored image based on a visible light image of the object captured using a visible light camera. The image processing device 10 according to this embodiment also calculates a second loss by identifying a colored mask image (second mask image) that has undergone mask processing based on a visible light mask image (first mask image) that has undergone mask processing using the map generated as described above. In this embodiment, the image generation unit 12 (coloring module 12b) includes a coloring model (first learning model) that has been trained to output a colored image in response to an input thermal infrared image, and the parameters of the coloring model are updated using the first and second losses calculated as described above.

[0089] In this embodiment, the image generation unit 12 (segmentation module 12a) uses a semantic segmentation model (second learning model) that has been trained to input a thermal infrared image and output semantic labels representing the types (meanings) of objects contained in the thermal infrared image to generate a semantic label map in which the semantic labels are assigned to each pixel constituting the thermal infrared image. The semantic segmentation model has an encoder-decoder structure, and the coloring model further inputs the features of the thermal infrared image extracted by the encoder of the semantic segmentation model and outputs a colored image.

[0090] One approach that could be considered here is to use a neural network to colorize a thermal infrared image and generate a pseudo-visible light image (color image). However, images generated using techniques such as simply using a neural network often have a semantically unnatural appearance, such as foreground objects blending into the background area, and there are still issues with image quality.

[0091] In contrast to this, in this embodiment, with the above-described configuration, by combining a coloring model with a network model for semantic segmentation, it becomes possible to generate a high-quality colored image by coloring a thermal infrared image.

[0092] Semantic segmentation is the task of dividing an image into regions according to the meaning of the objects. A model for this task (i.e., a semantic segmentation model) can be used to extract features that focus on the semantic information (semantic labels) of objects in a thermal infrared image, and by using these features as auxiliary input, it is possible to reinforce (strengthen) the semantic understanding of the coloring model for the thermal infrared image.

[0093] In addition, in this embodiment, an adversarial loss (first adversarial loss) is calculated as a first loss from the output of the whole image discrimination model to which a visible light image and a colored image are input, and a class adversarial loss (second adversarial loss) is calculated as a second loss from the output of the class image discrimination model to which a visible light mask image and a colored mask image are input.

[0094] For example, when only content loss is used, a colored image with blurred contours and an overall average color is likely to be generated, but in this embodiment, by using adversarial loss, it is possible to train a coloring model that outputs a realistic-looking colored image that retains detailed texture. Furthermore, by using a semantic label map to cut out areas to which specific semantic labels, such as cars or people, are assigned (areas belonging to a specific class), and using the class adversarial loss calculated only for the relevant areas, it is possible to train a coloring model that improves the local quality of the colored image.

[0095] The adversarial loss and class-adversarial loss described above may be calculated using an RSGAN. To prevent the classifier's authenticity determination from focusing solely on the quality of the colorized image, an RSGAN uses the relative authenticity between the colorized image (generated image) and the visible light image (ground-truth) as a loss. RSGAN is said to be able to utilize visible light image information more effectively than conventional GANs, and this characteristic is believed to improve the quality of colorized images when using an RSGAN.

[0096] Furthermore, in the coloring model of this embodiment, not only are the features of the thermal infrared image extracted in the semantic segmentation model input, but the semantic labels output from the semantic segmentation model are also combined in a combination layer included in the coloring model. This configuration makes it possible to achieve appropriate selection of colors for each semantic label (class) and consistency in coloring.

[0097] In this embodiment, the adversarial loss and the class adversarial loss are mainly calculated as the first and second losses. However, for example, a third loss may be further calculated based on a visible light image and a colored image, and a coloring model may be trained using the third loss. In this case, the third loss may include, for example, at least one of a content loss, a perceptual loss, and a total variation loss. With this configuration, the quality of the colored image can be expected to improve by training the coloring model using a composite loss.

[0098] Furthermore, in this embodiment, the coloring model used in the image generating unit 12 (generator) has been described as being a U-Net neural network, but the coloring model may be a network model based on another algorithm.

[0099] While the present embodiment has been described primarily with respect to training a coloring model (updating the parameters of the coloring model) to generate a high-quality colored image, the trained coloring model (i.e., the image processing device 10) can actually be used to utilize thermal infrared images in various fields. Specifically, when a thermal infrared image captured using a thermal infrared sensor under a specific environment (e.g., in bad weather or a low-light environment) is input to the input unit 11, the image generation unit 12 (a generator having a coloring module and a segmentation module) operates based on the thermal infrared image to generate a colored image from the thermal infrared image. The colored image thus generated may be transmitted to an external device, such as a server device, or displayed on a display device, such as a monitor.

[0100] The colored image generated using the coloring model that has been trained as described in this embodiment is an image that is close to a visible light image, and it is easy to interpret (understand) the objects contained in the thermal infrared image and the situation around the objects compared to the thermal infrared image. Therefore, it is considered to be useful when using thermal infrared images captured using a thermal infrared camera in various fields.

[0101] In this embodiment, a semantic segmentation model realized by a convolutional neural network is used, and a class mask is generated from a semantic label map based on the semantic labels output from the semantic segmentation model. However, the mask generated in this embodiment is not limited to the class mask.

[0102] A modified example of this embodiment will be described below with reference to Fig. 6. In Fig. 6, the same parts as those in Fig. 4 are given the same reference numerals, and detailed description thereof will be omitted.

[0103] 6, the image generation unit 12 (generator) may be configured to include an edge filter 12c instead of the segmentation module 12a (semantic segmentation model) described in this embodiment. With this configuration, for example, an edge map representing edges of objects included in a thermal infrared image extracted by the edge filter 12c that receives the thermal infrared image is generated, and a mask in which a bounding box encompassing the object is formed from the edge map is generated.

[0104] In the example shown in Fig. 6, the visible light mask image and the colored mask image that have undergone mask processing using such a mask are input to an instance image classifier 14c (instance image classification model) that is provided instead of the class image classifier 14b. According to this, the adversarial loss L ins and L Dins is calculated, and the hostile loss L ins and L Dins It is possible to train a coloring model using

[0105] Although not shown in FIG. 6, the coloring module 12b (coloring model) may be configured to input the edges output from the edge filter 12c and output a colored image.

[0106] The present invention is not limited to the above-described embodiments, and the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be created by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined. [Explanation of symbols]

[0107] 10...image processing device, 11...input unit, 12...image generation unit, 12a...segmentation module, 12b...coloring module, 12c...edge filter, 13...mask generation unit, 14...loss calculation unit, 14a...full image classifier, 14b...class image classifier, 14c...instance image classifier, 15...learning unit, 101...CPU, 102...non-volatile memory, 103...main memory, 104...communication device

Claims

1. an input means for inputting a thermal infrared image of an object captured using a thermal infrared camera; a first generating means for generating a colored image by coloring the input thermal infrared image from the input thermal infrared image; a second generating means for generating a map representing an object included in the input thermal infrared image from the input thermal infrared image; a first calculation means for calculating a first loss by discriminating the generated colored image based on a visible light image of the object captured using a visible light camera; a second calculation means for calculating a second loss by determining a second mask image obtained by performing the mask processing on the colored image based on a first mask image obtained by performing the mask processing on the visible light image using the generated map; Equipped with the first generation means includes a first learning model that is trained to output the colored image by inputting the thermal infrared image; The parameters of the first learning model are updated using the calculated first and second losses. Image processing device.

2. the second generation means uses a second learning model that has been trained to input the thermal infrared image and output semantic labels that represent types of objects included in the thermal infrared image to generate a semantic label map in which the semantic labels are assigned to each pixel that constitutes the thermal infrared image; the second training model has an encoder-decoder structure; The first learning model further inputs the feature amount of the thermal infrared image extracted by the encoder unit of the second learning model and outputs the colored image.

2. The image processing device according to claim 1.

3. The image processing device according to claim 2 , wherein the first learning model includes a connection layer that connects the semantic labels output from the second learning model.

4. 3. The image processing device according to claim 2, wherein the first learning model is a U-Net neural network.

5. The image processing device according to claim 2 , wherein the second learning model is a semantic segmentation model realized by a convolutional neural network.

6. the first calculation means calculates a first adversarial loss as the first loss from an output of a first discriminant model to which the visible light image and the colored image are input; The second calculation means calculates a second adversarial loss as the second loss from an output of a second discriminant model to which the first mask image and the second mask image are input.

2. The image processing device according to claim 1.

7. The image processing device according to claim 6 , wherein the first and second adversarial losses are calculated using a Relativistic Standard Generative Adversarial Network (RSGAN).

8. a third calculation means for calculating a third loss based on the visible light image and the colored image, The third loss includes at least one of a content loss, a perceptual loss, and a total variation loss.

7. The image processing device according to claim 6.

9. The image processing apparatus according to claim 1 , wherein the second generating means generates an edge map representing edges of objects included in the thermal infrared image.

10. A step of inputting a thermal infrared image of an object captured using a thermal infrared camera; generating a colored image by coloring the input thermal infrared image; generating a map representing objects contained in the input thermal infrared image from the input thermal infrared image; calculating a first loss by discriminating the generated colored image based on a visible light image captured by a visible light camera of the object; calculating a second loss by determining a second mask image obtained by performing the mask processing on the colored image based on a first mask image obtained by performing the mask processing on the visible light image using the generated map; Equipped with the colored image is generated using a first learning model that is trained to output the colored image by inputting the thermal infrared image; The parameters of the first learning model are updated using the calculated first and second losses. Image processing methods.

11. On the computer, A step of inputting a thermal infrared image of an object captured using a thermal infrared camera; generating a colored image by coloring the input thermal infrared image; generating a map representing objects contained in the input thermal infrared image from the input thermal infrared image; calculating a first loss by discriminating the generated colored image based on a visible light image captured by a visible light camera of the object; calculating a second loss by determining a second mask image obtained by performing the mask processing on the colored image based on a first mask image obtained by performing the mask processing on the visible light image using the generated map; Execute the colored image is generated using a first learning model that is trained to output the colored image by inputting the thermal infrared image; The parameters of the first learning model are updated using the calculated first and second losses. program.

Images