COLOR IMAGE COMPLETION METHOD AND NEURAL NETWORK LEARNING METHOD

Abstract

This color image complementing method comprises: a color image acquiring step (S10) for acquiring a color image of an object to be recognized including a missing portion where image information is missing; an infrared image acquiring step (S11) for acquiring an infrared image of the same object to be recognized, and generating an infrared image for processing; a missing portion identifying step (S12) for identifying the missing portion in the acquired color image; and a complementing step (S13) for complementing, with respect to the acquired color image, the missing portion identified in the missing portion identifying step (S12). In the complementing step (S13), the missing portion is complemented with the color image, using the information of the infrared image for processing corresponding to the missing portion.

Description

[Technical field]

[0001] The present disclosure relates to a color image completion method and a neural network training method, and to image processing for completing missing parts of a color image. [Background technology]

[0002] In order to prevent traffic accidents caused by vehicle drivers, systems have been devised that monitor the driver's situation and physical condition, and alert the driver and warn him / her. For example, a camera inside the vehicle is used to detect the driver's line of sight and detect whether the driver is dozing off. However, in systems that use color images obtained from a general camera, there are cases where the color images are missing due to the influence of headlights of oncoming vehicles or the driver wearing sunglasses, and as a result, the driver's line of sight cannot be confirmed.

[0003] Therefore, conventionally, techniques have been proposed for complementing missing parts of color images obtained by a camera (see, for example, Non-Patent Documents 1 to 3).

[0004] In Non-Patent Documents 1 and 2, weighting parameters are generated by comparing features such as contrast between a color image and an infrared image obtained by photographing the same object, and the color image and the infrared image are synthesized using the weighting parameters to perform image correction of edge components and contrast.

[0005] In addition, in non-patent document 3, noise (i.e., a mask) is intentionally introduced into a color image, and a machine-learned inference engine is used to restore the original color image from the color image with the noise introduced, thereby filling in missing parts in the color image from surrounding information.

[0006] In this specification, a color image is an image obtained by an image sensor having red, blue, and green pixels, and is also called an RGB image. An infrared image is an image obtained by an image sensor having pixels sensitive to infrared rays, and is also called an Ir image. A missing part is a part where image information is missing. In this specification, "machine learning" is also simply called "learning". [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Xiaopeng Zhang, et.al., “Enhancing photographs with Near Infra-Red images”, 2008 IEEE Conference on Computer Vision and Pattern Recognition (https: / / ieeexplore.ieee.org / document / 4587825) [Non-Patent Document 2] Ahmed Elliethy, et.al., “Fast near infrared fusion-based adaptive enhancement of visible images”, 2017 IEEE Global Conference on Signal and Information Processing (GlobalSIP) (https: / / ieeexplore.ieee.org / document / 8308623) [Non-Patent Document 3] Guilin Liu, et.al., “Image Inpainting for Irregular Holes Using Partial Convolutions”, ECCV 2018 (https: / / rd.springer.com / chapter / 10.1007 / 978-3-030-01252-6_6) Summary of the Invention [Problem to be solved by the invention]

[0008] However, in the technologies of Non-Patent Documents 1 and 2, the missing parts of the color image are simply complemented by the infrared image, so color information that cannot be obtained from the infrared image is not complemented. Therefore, even if the parts hidden by the driver's sunglasses are complemented, there is a problem that it is not possible to determine the degree of bloodshot eyes, the complexion of the driver (whether or not he / she is under the influence of alcohol), etc.

[0009] In addition, in the technology of Non-Patent Document 3, missing parts in a color image are filled from image information around the missing parts based on knowledge obtained by learning, so if a completely hidden part is filled, it will become a completely different image. Therefore, if the driver's eyes that are completely hidden by sunglasses are filled, they will be filled with the eyes of a different person.

[0010] Therefore, an object of the present disclosure is to provide a color image completion method and a neural network learning method that can complete missing parts of a color image with higher accuracy than conventional methods. [Means for solving the problem]

[0011] In order to achieve the above object, a color image completion method according to one embodiment of the present disclosure includes a color image acquisition step of acquiring a color image of a recognized object including a missing portion where image information is missing, an infrared image acquisition step of acquiring an infrared image of the recognized object and generating an infrared image for processing, a missing portion identification step of identifying the missing portion in the color image, and a completion step of completing the missing portion identified in the missing portion identification step in the color image, wherein in the completion step, the missing portion is completed in the color image using information of the infrared image for processing corresponding to the missing portion.

[0012] In order to achieve the above-mentioned object, a neural network learning method according to one embodiment of the present disclosure is a neural network learning method that infers a color image of a recognized object having a missing portion complemented from a color image of the recognized object, the color image including a missing portion where image information is missing, and includes a color image preparation step of preparing a color image of the recognized object, an infrared image preparation step of preparing an infrared image for processing of the recognized object, a missing portion information preparation step of preparing missing portion information indicating the missing portion, a masked color image preparation step of preparing a masked color image that is a color image including the missing portion by masking the color image with the missing portion information, and a learning step of inputting the masked color image, the infrared image for processing, and the missing portion information into the neural network, and training the neural network using the color image prepared in the color image preparation step as teacher data. Effect of the Invention

[0013] According to the present disclosure, a color image completion method and a neural network learning method are provided that can complete missing parts of a color image with higher accuracy than conventional methods. [Brief description of the drawings]

[0014] [Figure 1] FIG. 1 is a block diagram showing a configuration of a color image completion device according to an embodiment. [Diagram 2] FIG. 2 is a flowchart showing the operation of the color image completion device according to the embodiment. [Diagram 3] FIG. 3 is a block diagram showing a configuration of a neural network learning device that trains the inference engine included in the color image completion device according to the embodiment. [Figure 4] FIG. 4 is a flowchart showing the operation of the neural network learning device according to the embodiment. [Diagram 5]FIG. 5 shows an example of a color image (FIG. 5(a)) and a grayscale image (FIG. 5(b)) serving as a pseudo-infrared image of the same object to be recognized, which were used in the experiment to train the inference engine. [Figure 6] FIG. 6 is a diagram showing an example of completion by a color image completion device that has completed learning using the image shown in FIG. 5 in an experiment. [Figure 7] FIG. 7 is a diagram showing an example of complementation by a color image complementation method according to a comparative example. [Figure 8] FIG. 8 is a block diagram showing a configuration of a neural network learning device according to a first modified example of the embodiment. [Figure 9] FIG. 9 is a block diagram showing a configuration of a neural network learning device according to a second modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, the embodiments of the present disclosure will be described in detail with reference to the drawings. Each of the embodiments described below shows a specific example of the present disclosure. The numerical values, shapes, materials, components, the arrangement and connection of the components, steps, and the order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, each figure is not necessarily a strict illustration. In each figure, the same reference numerals are used for substantially the same configurations, and duplicated explanations are omitted or simplified.

[0016] 1 is a block diagram showing a configuration of a color image completion device 10 according to an embodiment. The color image completion device 10 is a device that completes a missing portion in a color image of a recognition target object including the missing portion with a color image, and includes a color image acquisition unit 11, an infrared image acquisition unit 12, a mask generation unit 13, a mask calculation unit 14, a filter unit 15, a combination unit 16, and an inference engine 20. Note that the color image completion device 10 is typically realized by a program executed by a processor. The program can be stored in a non-transitory recording medium such as a computer-readable DVD.

[0017] The color image acquisition unit 11 acquires a color image of the object to be recognized including a missing portion. For example, the color image acquisition unit 11 acquires a color image from a camera having an RGB image sensor, or from an auxiliary storage device such as a hard disk. The missing portion is a portion where image information is missing, and includes a foreign object (for example, sunglasses covering the subject or dust on the lens), whiteout or blackout due to uneven exposure, a portion where a desired image cannot be obtained due to a defect in a part of the RGB image sensor, etc.

[0018] The infrared image acquisition unit 12 acquires an infrared image of the same object to be recognized as the object to be recognized in the color image acquired by the color image acquisition unit 11. For example, the color image acquisition unit 11 acquires an infrared image from a camera having an infrared image sensor or from an auxiliary storage device such as a hard disk. Note that the infrared image contains information that cannot be obtained from a color image, such as the outline of the eyes hidden by sunglasses.

[0019] The mask generation unit 13 generates a mask, which is map data indicating missing parts in the color image, from the color image acquired by the color image acquisition unit 11 and the infrared image acquired by the infrared image acquisition unit 12. At this time, the mask generation unit 13 generates the mask taking into consideration at least one of the saturation and luminance of the color image, and at least one of the contrast and sharpness of the infrared light relative to the color image.

[0020] More specifically, the mask generating unit 13 calculates a weight W from the color image acquired by the color image acquiring unit 11, as shown in the following formula 1, which is larger for areas with lower saturation and brightness.

[0021]

number

[0022] where s(x) and v(x) are the saturation and brightness of the color image at the coordinate x, respectively, and p s(s(x)), p v (v(x)) are the frequencies obtained from the saturation and luminance histograms, respectively.

[0023] Then, the mask generation unit 13 calculates a weight F that increases the value of areas where the contrast and sharpness of the color image acquired by the color image acquisition unit 11 is low compared to the infrared image acquired by the infrared image acquisition unit 12, as shown in the following equation 2.

[0024]

number

[0025] Here, I NIR (x) is the brightness at coordinate x in the infrared image. RGB_Y (x) = v(x).

[0026] Then, the mask generation unit 13 generates a map M that indicates that the weighted sum itself, based on the weight W calculated by the above formula 1 and the weight F calculated by the above formula 2, is valid (i.e., that it is a missing part) when the weighted sum reaches or exceeds a certain value, as shown in the following formula 3, or that the weighted sum is valid (i.e., that it is a missing part) when it reaches or exceeds a certain value, as shown in the following formula 4.

[0027]

number

[0028]

number

[0029] The mask calculation unit 14 generates a masked color image by performing mask processing on the color image acquired by the color image acquisition unit 11 using the mask generated by the mask generation unit 13. The masked color image is, for example, a color image in which the pixel values ​​of the masked missing parts are zero (black) and the pixel values ​​of the pixels other than the missing parts remain the same as in the original color image.

[0030] The filter unit 15 extracts high-frequency components from the infrared image acquired by the infrared image acquisition unit 12, and generates an image of the extracted high-frequency components as an infrared image for processing. The high-frequency components of the infrared image are extracted in order to acquire information that is highly correlated with the color image. Specifically, the filter unit 15 performs a filter process called a LoG (Laplacian Of Gaussian) filter, which is a combination of a Gaussian filter and a Laplacian filter, or a filter process called a DoG (Difference Of Gaussian) filter, which creates a difference image (DoG image) of two Gaussian filter images with different standard deviation σ values, according to a pre-set setting. Note that the LoG filter smoothes the image with a Gaussian filter to reduce noise, and then extracts the contour with a Laplacian filter. The Laplacian filter has the characteristic of easily emphasizing noise because it works as a secondary derivative. On the other hand, the DoG filter is used instead of the LoG filter because the processing is similar to that of the LoG filter but the amount of calculation is small.

[0031] The combining unit 16 combines the masked color image generated by the mask calculation unit 14 with the infrared image for processing generated by the filter unit 15. Specifically, the combining unit 16 combines the masked color image with the infrared image for processing, and outputs an image having a data structure that matches the configuration of the input layer of the inference engine 20.

[0032] The inference engine 20 is a learning model that is trained to output a color image (complemented color image) in which the missing parts are complemented with a color image from a masked color image including missing parts. Specifically, the inference engine 20 is a U-shaped convolutional neural network (i.e., a deep learning model) called U-Net, and is composed of downward convolutional layers 21a to 21g, upward convolutional layers 22a to 22h, and merge layers 23a to 23c. In the downward convolutional layers 21a to 21g, convolution and pooling are performed, and the deeper the layer, the more localized the image features are and the smaller the image size becomes. On the other hand, in the upward convolutional layers 22a to 22h, convolution and upsampling are performed, and the image size is restored to a large size while retaining the features. In the merge layers 23a to 23c, images of the same size are merged in stages from the deeper layers. In the convolution, a method called "Partial Convolution" is used to convey information about the missing parts while convolution is performed (see Non-Patent Document 3).

[0033] FIG. 2 is a flowchart showing the operation of the color image completion device 10 according to the present embodiment configured as above (that is, the color image completion method).

[0034] First, the color image acquisition unit 11 acquires a color image of the recognition target object including the missing portion (color image acquisition step S10).

[0035] Next, the color image completion device 10 acquires an infrared image of the same object to be recognized as the object to be recognized in the color image by the infrared image acquisition unit 12, extracts high-frequency components from the acquired infrared image using a LoG filter or a DoG filter by the filter unit 15, and generates an image of the extracted high-frequency components as an infrared image to be processed (infrared image acquisition step S11). Note that in this infrared image acquisition step S11, the infrared image acquired by the infrared image acquisition unit 12 may be generated as it is as an infrared image to be processed by bypassing the filter unit 15. Also, the infrared image acquisition step S11 may be performed before the color image acquisition step S10, or may be performed in parallel with the color image acquisition step S10.

[0036] Next, the mask generating unit 13 identifies missing parts in the color image acquired by the color image acquiring unit 11 (missing part identifying step S12). Specifically, the mask generating unit 13 generates a mask, which is map data indicating missing parts in the color image, from the color image acquired by the color image acquiring unit 11 and the infrared image acquired by the infrared image acquiring unit 12. At this time, the mask generating unit 13 calculates a weight W from the color image acquired by the color image acquiring unit 11, which is larger in areas with low saturation and brightness, calculates a weight F that is larger in areas with low contrast and sharpness in the color image acquired by the color image acquiring unit 11 relative to the infrared image acquired by the infrared image acquiring unit 12, and generates a map M indicating the missing parts as in the above formula 3 or 4 using a weighted sum of the calculated weight W and the calculated weight F.

[0037] Then, the color image completion device 10 uses the mask calculation unit 14, the combining unit 16, and the inference engine 20 to complete the missing portion identified in the missing portion identification step S12 for the color image acquired by the color image acquisition unit 11, using information on the infrared image for processing corresponding to the missing portion (completion step S13). Specifically, the mask calculation unit 14 first performs mask processing on the color image acquired by the color image acquisition unit 11, using the mask generated by the mask generation unit 13, to generate a masked color image. Then, the combining unit 16 combines the masked color image generated by the mask calculation unit 14 with the infrared image for processing generated by the filter unit 15. Furthermore, the inference engine 20 generates a color image in which the missing portion is completed with the color image (i.e., a completed color image) from the masked color image including the missing portion.

[0038] Finally, the inference engine 20 outputs the generated complemented color image (output step S14).

[0039] As described above, the color image completion method according to the present embodiment includes a color image acquisition step S10 for acquiring a color image of a recognized object including a missing portion where image information is missing, an infrared image acquisition step S11 for acquiring an infrared image of the same recognized object and generating an infrared image for processing, a missing portion identification step S12 for identifying the missing portion in the acquired color image, and an completion step S13 for completing the missing portion identified in the missing portion identification step S12 for the acquired color image, where the missing portion is completed in the color image using information of the infrared image for processing corresponding to the missing portion. Note that although information indicating the missing portion is used in step S13, information indicating the missing portion identified in step S12 may also be used.

[0040] As a result, missing parts of a color image are correctly restored in color image using infrared images that can obtain information that cannot be obtained from visible light images. Therefore, unlike the techniques of Non-Patent Documents 1 and 2, for example, parts hidden by the driver's sunglasses are complemented with color images, and even the degree of bloodshot eyes and the complexion of the driver (whether or not he / she is under the influence of alcohol) can be determined. Also, unlike the technique of Non-Patent Document 3, an infrared image corresponding to the missing parts is also used, so that the problem of completely hidden parts being complemented with a completely different image is avoided. Thus, a color image complementing method that can complement missing parts of a color image with higher accuracy than conventional methods is realized.

[0041] Here, in the infrared image acquisition step S11, an image in which high-frequency components of the infrared image are extracted is generated as an infrared image for processing. As a result, the high-frequency components of the infrared image are used to complement the missing parts, rather than the infrared image itself, so that the difference in image characteristics caused by the difference in the reflection characteristics of the visible light used to generate the color image and the near-infrared light used to generate the infrared image is mitigated, and the missing parts are complemented with high restoration power.

[0042] In the missing portion identifying step S12, the missing portion is identified taking into consideration at least one of the saturation and luminance of the color image, and at least one of the contrast and sharpness of the infrared image relative to the color image. In this way, the missing portion is identified using the characteristics of both the color image and the infrared image, so that the portion where image information is missing, which is different from the surroundings, can be identified with high accuracy.

[0043] Next, a learning method for the inference engine 20 in the above embodiment (that is, a neural network learning method) will be described.

[0044] 3 is a block diagram showing the configuration of a neural network learning device 30 that trains the inference engine 20 included in the color image completion device 10 according to the above embodiment. The inference engine 20 to be trained is also shown in the figure.

[0045] The neural network training device 30 is a device that trains the inference engine 20, and in addition to the color image acquisition unit 11, infrared image acquisition unit 12, mask calculation unit 14, filter unit 15, and combination unit 16 that are provided in the color image completion device 10 of the above embodiment, it also includes a training mask generation unit 31, a luminance generation unit 32, a selection unit 33, and a loss calculation unit 34.

[0046] The color image acquisition unit 11 acquires a color image of the object to be recognized as input data for learning. For example, the color image acquisition unit 11 acquires a color image for learning from an auxiliary storage device such as a hard disk. Note that this color image for learning is a color image of the object to be recognized that does not include missing parts, and is, for example, a color image obtained by photographing a driver not wearing sunglasses with a camera under normal conditions. The color image acquired by the color image acquisition unit 11 is output to the mask calculation unit 14, the brightness generation unit 32, and the loss calculation unit 34.

[0047] The learning mask generating unit 31 generates a mask used to generate the missing parts, that is, missing part information indicating the missing parts. For example, the learning mask generating unit 31 regards the color image acquired by the color image acquiring unit 11 as the whole, randomly generates a group of partial areas from the whole, and generates a mask indicating the generated partial areas as missing parts as missing part information.

[0048] The mask calculation unit 14 generates a masked color image, which is a color image including missing parts, by masking the color image acquired by the color image acquisition unit 11 with the missing part information generated by the learning mask generation unit 31.

[0049] The luminance generating unit 32 generates a grayscale image from the color image acquired by the color image acquiring unit 11. Specifically, the luminance generating unit 32 generates a grayscale image by extracting only the luminance component from the color image acquired by the color image acquiring unit 11. This grayscale image is used as a substitute for an infrared image (i.e., a pseudo infrared image) when it is difficult to prepare an infrared image.

[0050] The infrared image acquisition unit 12 acquires an infrared image of the same object to be recognized as the object to be recognized in the color image. For example, the color image acquisition unit 11 acquires an infrared image of the object to be recognized from an auxiliary storage device such as a hard disk.

[0051] The selection unit 33 selects either the grayscale image generated by the luminance generation unit 32 or the infrared image acquired by the infrared image acquisition unit 12 according to a previous setting, and outputs the selected image to the filter unit 15. For example, when the infrared image acquired by the infrared image acquisition unit 12 is not stored in the auxiliary storage device, the selection unit 33 selects the grayscale image generated by the luminance generation unit 32, whereas when the infrared image acquired by the infrared image acquisition unit 12 is stored in the auxiliary storage device, the selection unit 33 selects the infrared image. The selection unit 33 is provided to enable the inference engine 20 to learn using either the infrared image or a grayscale image that replaces the infrared image.

[0052] The filter unit 15 extracts high-frequency components from the grayscale image or infrared image output from the selection unit 33, and generates an image of the extracted high-frequency components as an infrared image to be processed. Specifically, the filter unit 15 performs filtering using a LoG filter or a DoG filter on the grayscale image or infrared image output from the selection unit 33 in accordance with a previous setting, and outputs the obtained image as the infrared image to be processed.

[0053] The combining unit 16 combines the masked color image generated by the mask calculation unit 14 with the infrared image for processing generated by the filter unit 15. Specifically, the combining unit 16 combines the masked color image with the infrared image for processing, outputs an image having a data structure that matches the configuration of the input layer of the inference engine 20, and inputs it to the inference engine 20.

[0054] The loss calculation unit 34 uses the color image acquired by the color image acquisition unit 11 as teaching data and causes the inference engine 20 to perform supervised learning. Specifically, the loss calculation unit 34 adjusts the model parameters of the inference engine 20 by backpropagation, stochastic gradient descent, or the like so that the complemented color image output by the inference engine 20 becomes closer to the color image acquired by the color image acquisition unit 11.

[0055] FIG. 4 is a flowchart showing the operation of the neural network learning device 30 according to the present embodiment configured as above (that is, the neural network learning method).

[0056] First, the color image acquisition unit 11 prepares a color image of the object to be recognized as input data for learning (color image preparation step S20). For example, the color image acquisition unit 11 prepares the color image for learning by acquiring it from an auxiliary storage device such as a hard disk. The prepared color image is a color image of the object to be recognized that does not include missing parts.

[0057] Next, the neural network learning device 30 uses the luminance generating unit 32, the infrared image acquiring unit 12, and the selecting unit 33 to prepare an infrared image for processing of the same object to be recognized as the object to be recognized in the learning color image (infrared image preparing step S21). More specifically, the luminance generating unit 32 generates a grayscale image from the color image acquired by the color image acquiring unit 11. Meanwhile, the infrared image acquiring unit 12 acquires an infrared image of the object to be recognized. Then, the selecting unit 33 selects either the grayscale image generated by the luminance generating unit 32 or the infrared image acquired by the infrared image acquiring unit 12 according to a previous setting, and outputs the selected image to the filter unit 15. Furthermore, the filter unit 15 extracts high-frequency components by applying a filter process using a LoG filter or a DoG filter to the grayscale image or the infrared image output from the selecting unit 33, and prepares the image of the extracted high-frequency components as an infrared image for processing.

[0058] Then, the learning mask generating unit 31 prepares a mask used to generate the missing parts, that is, missing part information indicating the missing parts (missing part information preparation step S22). The missing part information is, for example, mask information that indicates a group of partial regions in the entire color image acquired by the color image acquiring unit 11 as a missing part.

[0059] Next, the mask calculation unit 14 prepares a masked color image, which is a color image including missing parts, by masking the color image acquired by the color image acquisition unit 11 with the missing part information generated by the learning mask generation unit 31 (masked color image preparation step S23).

[0060] Then, the neural network learning device 30 inputs the masked color image prepared in the masked color image preparation step S23, the infrared image for processing prepared in the infrared image preparation step S21, and the missing part information prepared in the missing part information preparation step S22 to the inference engine 20, and trains the inference engine 20 using the color image prepared in the color image preparation step S20 as training data (training step S24). More specifically, the combining unit 16 generates an image having a data structure that matches the configuration of the input layer of the inference engine 20 by combining the masked color image generated by the mask calculation unit 14 and the infrared image for processing generated by the filter unit 15, and inputs the generated image and the missing part information generated by the learning mask generation unit 31 to the inference engine 20. Then, the loss calculation unit 34 adjusts the model parameters of the inference engine 20 using the color image acquired by the color image acquisition unit 11 as training data so that the complemented color image output by the inference engine 20 is close to the training data.

[0061] Through the above procedure, the neural network learning device 30 learns the inference engine 20.

[0062] Next, an experiment performed on the operation of the color image completion device 10 according to the embodiment will be described. FIG. 5 is a diagram showing an example of a color image (FIG. 5(a)) and a grayscale image (FIG. 5(b)) serving as a pseudo infrared image of the same object to be recognized, which were used in the experiment for training the inference engine 20. Note that the images shown in FIG. 5 are schematic diagrams in which the images actually used have been simplified to black and white images for convenience of illustration. The same applies to FIGS. 6 and 7, which will be described later. During the training experiment, of the infrared images and grayscale images serving as pseudo infrared images input to the selection unit 33, only the grayscale image serving as the pseudo infrared image was used.

[0063] FIG. 6 shows an example of completion by the color image completion device 10 that has completed learning using the image shown in FIG. 5 in an experiment. More specifically, FIG. 6(a) shows an example of a color image including a missing portion that is input to the color image completion device 10. The missing portion is an area of ​​sunglasses on a person's face. FIG. 6(b) shows a completed color image output by the color image completion device 10. In the completed color image shown in FIG. 6(b), the missing portion has been restored to a color image that is close to the true image.

[0064] FIG. 7 is a diagram showing an example of complementation by a color image complementation method according to a comparative example using the method disclosed in Non-Patent Document 3 (i.e., a method of complementing missing parts of a color image using only a color image without using an infrared image). More specifically, FIG. 7(a) shows an example of a color image including missing parts input in the color image complementation method according to the comparative example, which is the same as that shown in FIG. 6(a). FIG. 7(b) shows a complemented color image output in the color image complementation method according to the comparative example. In the complemented color image shown in FIG. 7(b), as can be seen by comparing with FIG. 6(b), the missing parts are restored with a rough image and are not restored correctly.

[0065] As described above, the color image completion method according to the present embodiment includes a color image acquisition step S10 for acquiring a color image of a recognized object including a missing portion where image information is missing, an infrared image acquisition step S11 for acquiring an infrared image of the same recognized object and generating a processing infrared image, a missing portion specification step S12 for specifying the missing portion in the acquired color image, and a completion step S13 for completing the missing portion specified in the missing portion specification step S12 for the acquired color image, and in the completion step S13, the missing portion is completed with a color image using information of the processing infrared image corresponding to the missing portion. Here, in the infrared image acquisition step S11, as an example, the infrared image is generated as it is as the processing infrared image, and in the completion step S13, the missing portion is completed using the learning color image, the learning infrared image corresponding to the learning color image, and the learning missing portion information, and the inference engine 20 that has been trained to infer the original learning color image from the learning color image with missing portion generated from the learning color image and the learning missing portion information and the learning infrared image is used.

[0066] In this way, an inference engine 20 is used that has been trained to infer the original training color image from the training color image with missing parts and the training infrared image, so that a color image in which the missing parts are complemented with the color image is restored from the color image including the missing parts.

[0067] Furthermore, in the complementation step S13, the missing parts may be complemented using an inference engine trained to infer the original training color image from the training color image with missing parts generated from the training color image and the training infrared image corresponding to the training color image, a training processing infrared image in which high-frequency components have been extracted, and training missing part information, and the training color image and the training processing infrared image.

[0068] As a result, the infrared image for learning processing is an image in which high-frequency components have been extracted from the learning infrared image, so differences in image features caused by the different reflection characteristics between visible light used to generate color images and near-infrared light used to generate infrared images are mitigated and missing parts are complemented with high restoration power.

[0069] Furthermore, the neural network learning method according to this embodiment is a learning method for a neural network (i.e., inference engine 20) that infers a color image of an object to be recognized that has missing parts filled in from a color image of the object to be recognized, including missing parts where image information is missing, and includes a color image preparation step S20 for preparing a color image of the object to be recognized, an infrared image preparation step S21 for preparing an infrared image for processing of the object to be recognized, a missing part information preparation step S22 for preparing missing part information indicating the missing part, a masked color image preparation step S23 for preparing a masked color image that is a color image including the missing part by masking the color image with the missing part information, and a learning step S24 for inputting the masked color image, the infrared image for processing, and the missing part information into the neural network and training the neural network using the color image prepared in the color image preparation step S20 as training data.

[0070] This allows a neural network (i.e., the inference engine 20) to learn how to restore a color image in which the missing parts are complemented with a color image from a color image including the missing parts. Therefore, the color image complementing device 10 using the thus trained neural network (i.e., the inference engine 20) complements the missing parts with a color image, unlike the techniques of Non-Patent Documents 1 and 2. For example, the part hidden by the driver's sunglasses is complemented with a color image, and it is possible to determine the degree of bloodshot eyes and the face color of the driver (whether or not the driver is under the influence of alcohol). Also, unlike the technique of Non-Patent Document 3, an infrared image corresponding to the missing parts is also used, so that the problem of completely hidden parts being complemented with a completely different image is avoided. Thus, a color image complementing device 10 that can complement missing parts of a color image with higher accuracy than before is realized.

[0071] Here, in the infrared image preparation step S21, an infrared image for processing may be prepared by photographing the object to be recognized with an infrared camera. This allows the use of an infrared image that can obtain information that cannot be obtained from a visible light image, such as the outline of an eye hidden by sunglasses, and therefore allows the acquisition of a neural network (i.e., the inference engine 20) that can correctly restore missing parts of a color image with a color image.

[0072] In addition, in the infrared image preparation step S21, a grayscale image may be generated from a color image, and then a high-frequency component of the grayscale image may be extracted to generate an infrared image for processing. This allows the neural network (i.e., the inference engine 20) to be properly trained using the grayscale image as a pseudo infrared image in place of an infrared image, even if it is difficult to prepare an infrared image.

[0073] Note that the neural network learning device 30 in the above embodiment is provided with a configuration (brightness generation unit 32, infrared image acquisition unit 12, and selection unit 33) that selectively uses an infrared image and a pseudo infrared image, but instead of these, it may be provided with only the brightness generation unit 32 and always use a pseudo infrared image, or instead of these, it may be provided with only the infrared image acquisition unit 12 and always use an infrared image.

[0074] Fig. 8 is a block diagram showing the configuration of a neural network learning device 30a according to a first modified example of the embodiment, which includes only a luminance generation unit 32 instead of the luminance generation unit 32, the infrared image acquisition unit 12, and the selection unit 33 in the neural network learning device 30 shown in Fig. 3. Even with this neural network learning device 30a, it is possible to properly train the inference engine 20 using a pseudo infrared image. In the learning in the above experiment, learning was performed with such a neural network learning device 30a.

[0075] Fig. 9 is a block diagram showing the configuration of a neural network learning device 30b according to a second modified example of the embodiment, which includes only an infrared image acquisition unit 12 instead of the luminance generation unit 32, the infrared image acquisition unit 12, and the selection unit 33 in the neural network learning device 30 shown in Fig. 3. Even with this neural network learning device 30b, it is possible to appropriately train the inference engine 20 using an infrared image.

[0076] The color image completion method and neural network learning method of the present disclosure have been described above based on the embodiments and modifications, but the present disclosure is not limited to these embodiments and modifications. As long as they do not deviate from the gist of the present disclosure, the present disclosure also includes various modifications that a person skilled in the art may make to the embodiments and modifications, and other forms constructed by combining some of the components of the embodiments and modifications.

[0077] For example, in the above embodiment, the color image completion device 10 and the neural network learning device 30 are provided with the filter unit 15, but they do not necessarily need to be provided with the filter unit 15. The filter unit 15 is intended to reduce differences in characteristics in the images caused by differences in the reflection characteristics of visible light used to generate a color image and near-infrared light used to generate an infrared image, and depending on the environment in which the color image and the infrared image are captured by the camera, there may be cases in which these differences in characteristics are small, that is, there may be cases in which the need for the filter unit 15 is low.

[0078] In the above embodiment, the mask generating unit 13 generates a mask using the saturation and brightness of the color image acquired by the color image acquiring unit 11 and the contrast and sharpness of the infrared image acquired by the infrared image acquiring unit 12, but this is not limited thereto, and the mask may be generated using at least one of the saturation and brightness of the color image and at least one of the contrast and sharpness of the infrared image relative to the color image. This is because, depending on the color of the object to be recognized and the environment in which the color image and the infrared image are acquired by the camera, there may be cases where a mask that accurately shows missing parts in the color image can be generated by using one of the saturation and brightness of the color image and one of the contrast and sharpness of the infrared image relative to the color image.

[0079] In the above embodiment, the color image completion device 10 is typically realized by a program executed by a processor, but the present invention is not limited to this and may be realized by hardware such as an electronic circuit. For example, the inference engine 20 may be realized by a field programmable gate array. [Industrial Applicability]

[0080] The color image completion method and neural network learning method disclosed herein can be used as a color image completion device that can complete missing parts of a color image with higher accuracy than conventional methods, for example, as a safe driving maintenance system that completes parts hidden by the driver's sunglasses and determines the degree of eye bloodshotness, the driver's complexion, etc. [Explanation of symbols]

[0081] 10. Color image completion device 11 Color image acquisition section 12 Infrared image acquisition unit 13 Mask generation unit 14 Mask calculation unit 15 Filter section 16 Joint 20 Inference Engine 21a~21g Downward convolution layer 22a~22h Upward convolution layer 23a-23c Merge layer 30, 30a, 30b Neural network learning device 31 Learning mask generator 32 Luminance generation unit 33 Selection section 34 Loss Calculation Section

Claims

1. A color image acquisition step of acquiring a color image of the object to be recognized including a missing portion where image information is missing; an infrared image acquisition step of acquiring an infrared image of the object to be recognized and generating an infrared image for processing; a missing portion identifying step of identifying the missing portion in the color image; a complementation step of complementing the missing portion identified in the missing portion identification step with respect to the color image, In the missing portion identification step, a mask generation unit that generates a mask, which is map data indicating the missing portion in the color image, calculates a first weight from the color image acquired in the color image acquisition step, the first weight being larger in areas of the color image with lower saturation and / or brightness, and calculates a second weight for the infrared image acquired in the infrared image acquisition step, the second weight being larger in areas of the color image acquired in the color image acquisition step with lower contrast and / or sharpness, and generates a map indicating the missing portion using a weighted sum of the calculated first weight and the calculated second weight. In the infrared image acquisition step, an image obtained by extracting high frequency components from the infrared image is generated as the processing infrared image; In the complementing step, a learning color image, a learning infrared image for processing in which high frequency components are extracted from a learning infrared image corresponding to the learning color image, and learning missing portion information are used, and the missing portion is complemented using an inference engine trained to infer the original learning color image from the learning color image and the learning color image with missing portions generated from the learning color image and the learning missing portion information, and the learning infrared image for processing; a mask calculation unit performs mask processing on the color image acquired in the color image acquisition step using the mask generated by the mask generation unit to generate a masked color image, and a combination unit combines the masked color image generated by the mask calculation unit with the infrared image for processing and outputs the combined image to an input layer of the inference engine, and the inference engine generates a complemented color image in which the missing portion is complemented with a color image from the masked color image including the missing portion. Color image completion method.

2. A color image acquisition step of acquiring a color image of the object to be recognized including a missing portion where image information is missing; an infrared image acquisition step of acquiring an infrared image of the object to be recognized and generating an infrared image for processing; a missing portion identifying step of identifying the missing portion in the color image; a complementation step of complementing the missing portion identified in the missing portion identification step with respect to the color image, In the missing portion identification step, a mask generation unit that generates a mask, which is map data indicating the missing portion in the color image, calculates a first weight from the color image acquired in the color image acquisition step, the first weight being larger in areas of the color image with lower saturation and / or brightness, and calculates a second weight for the infrared image acquired in the infrared image acquisition step, the second weight being larger in areas of the color image acquired in the color image acquisition step with lower contrast and / or sharpness, and generates a map indicating the missing portion using a weighted sum of the calculated first weight and the calculated second weight. In the infrared image acquisition step, the infrared image is generated as the processing infrared image as it is, In the complementing step, a learning color image, a learning infrared image corresponding to the learning color image, and learning missing portion information are used to complement the missing portion using an inference engine trained to infer the original learning color image from the learning infrared image and a learning color image with missing portions generated from the learning color image and the learning missing portion information, a mask calculation unit performs mask processing on the color image acquired in the color image acquisition step using the mask generated by the mask generation unit to generate a masked color image, and a combination unit combines the masked color image generated by the mask calculation unit with the infrared image for processing and outputs the combined image to an input layer of the inference engine, and the inference engine generates a complemented color image in which the missing portion is complemented with a color image from the masked color image including the missing portion. Color image completion method.

3. A neural network learning method for inferring a lossless color image of a recognition target object by complementing a lossy portion from a lossy color image of the recognition target object, the lossy portion being a portion where image information is missing, the method comprising: a color image preparation step of preparing a complete color image of the object to be recognized; an infrared image preparation step of preparing an infrared image for processing of the object to be recognized; a missing portion information preparation step of preparing missing portion information indicating the missing portion; a masked color image preparation step of preparing a masked color image, which is a color image including the missing portion, by masking the non-missing color image with the missing portion information; a learning step of inputting the masked color image, the infrared image for processing, and the missing portion information into the neural network, and training the neural network using the missing portion-free color image prepared in the color image preparation step as training data. Neural network training methods.

4. In the infrared image preparation step, the object to be recognized is photographed by an infrared camera to prepare the infrared image for processing.

4. The neural network training method according to claim 3.

5. In the infrared image preparation step, a grayscale image is generated from the lossless color image, and a high frequency component of the grayscale image is extracted to generate the infrared image for processing.

4. The neural network training method according to claim 3.

Images