Machine learning device and image processing device

Abstract

The far infrared image training data acquisition section (20) acquires far infrared images for a first prescribed period of time. The visible light image training data acquisition section (30) acquires visible light images for a second prescribed period of time. The visible light image generation model learning section (40) performs machine learning in a generative adversarial network using the far infrared images for the first prescribed period of time and the visible light images for the second prescribed period of time as training data, and generates a learned generation model (60) that generates visible light images for the second prescribed period of time from far infrared images for the first prescribed period of time. The visible light image generation model learning section (40) also generates a learned recognition model (70) that recognizes whether a far infrared image is one taken in the first prescribed period of time by performing machine learning in the generative adversarial network.

Description

Technical Field

[0001] This invention relates to a technique for converting far-infrared images into visible light images. Background Technology

[0002] Visible light cameras are less effective in fog or haze, and cannot photograph people and animals at night without a light source. On the other hand, far-infrared cameras are resistant to fog or haze and can photograph people and animals blurry even at night or in places without a light source, but it is difficult to identify people and animals.

[0003] In the absence of visible light at night, infrared cameras are used instead of visible light cameras to photograph objects. However, because far-infrared images lack color image information, it is difficult to detect specific objects such as people from them. Therefore, to make these objects easily recognizable by the human eye, a process is performed to convert the far-infrared images into visible light images.

[0004] Patent document 1 discloses an image processing apparatus in which a generator predicts the color of a subject based on the pixel values ​​of an infrared image, thereby generating a color image, and a control unit determines candidate colors based on the reliability of each color associated with the prediction.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent document 1: Japanese Patent Application Publication No. 2016-122905. Summary of the Invention

[0008] Converting infrared images to visible light images requires accurately assigning color values ​​to the pixel values ​​of the infrared image, which has limitations in terms of accuracy. Furthermore, it is difficult to convert nighttime infrared images into daytime color images.

[0009] The present invention was made in view of the following situation, and its object is to provide a technology capable of converting far-infrared images into visible light images with high precision.

[0010] To address the aforementioned issues, a machine learning apparatus according to a certain aspect of the present invention includes: a far-infrared image training data acquisition unit for acquiring far-infrared images over a first predetermined time period; a visible light image training data acquisition unit for acquiring visible light images over a second predetermined time period; and a visible light image generation model learning unit that uses the far-infrared images over the first predetermined time period and the visible light images over the second predetermined time period as training data, performs machine learning in a generative adversarial network, and generates a learned generative model that generates visible light images over the second predetermined time period based on the far-infrared images over the first predetermined time period.

[0011] Another aspect of the present invention is an image processing apparatus. This apparatus includes: a far-infrared image acquisition unit for acquiring far-infrared images; a visible light image generation unit for inputting the acquired far-infrared image into a generation model to generate a visible light image for a second predetermined time period, the generation model being obtained by performing machine learning on a generative adversarial network using far-infrared images for a first predetermined time period and visible light images for a second predetermined time period as training data; and a far-infrared image recognition unit for inputting the acquired far-infrared image into a recognition model generated by machine learning in the generative adversarial network, the recognition model being used to identify whether the far-infrared image was captured during the first predetermined time period. The visible light image generation unit generates the visible light image if the far-infrared image recognition unit determines that the far-infrared image was captured during the first predetermined time period; otherwise, it does not generate the visible light image if the far-infrared image was not captured during the first predetermined time period.

[0012] Another aspect of the present invention is a machine learning device. This device includes: a far-infrared image acquisition unit for acquiring far-infrared images; a visible light image generation unit for inputting the acquired far-infrared images into a generation model to generate visible light images over a second predetermined time period, wherein the generation model is obtained by performing machine learning in a generative adversarial network using far-infrared images over a first predetermined time period and visible light images over the second predetermined time period as training data; an object detection model learning unit for performing machine learning on an object detection model using the generated visible light images as training data; and a visible light image recognition unit for inputting the generated visible light images into a recognition model generated by performing machine learning in a generative adversarial network, wherein the recognition model identifies whether the visible light images are normal. The object detection model learning unit uses the generated visible light images as training data when the visible light image recognition unit determines that the generated visible light images are normal, and does not use the generated visible light images as training data when the object detection model learning unit determines that the generated visible light images are abnormal.

[0013] Furthermore, any combination of the above-mentioned constituent elements and any transformation of the present invention into methods, apparatus, systems, recording media, computer programs, etc., are also valid embodiments of the present invention.

[0014] According to the present invention, far-infrared images can be converted into visible light images with high precision. Attached Figure Description

[0015] Figure 1 This is a structural diagram of the machine learning device involved in Implementation Method 1.

[0016] Figure 2 This is a structural diagram of the far-infrared imaging device according to Embodiment 1.

[0017] Figure 3 (a) and (b) are diagrams illustrating the specified time period.

[0018] Figure 4 This is to explain the passage. Figure 1 The flowchart shows the steps of using a machine learning device to perform machine learning on the generative model of a visible light image generation network and the recognition model of a far-infrared image recognition network.

[0019] Figure 5 This is to explain the passage. Figure 2 A flowchart illustrating the steps of converting a far-infrared image into a visible light image using a visible light image generation network's generation model and a far-infrared image recognition network's recognition model in an image processing device.

[0020] Figure 6 (a) and (b) are diagrams illustrating the specified time periods after segmentation.

[0021] Figure 7 This is a structural diagram of the machine learning device involved in Implementation Method 4.

[0022] Figure 8 This is a structural diagram of the far-infrared imaging device according to Embodiment 4.

[0023] Figure 9 This is to explain the passage. Figure 7 The flowchart shows the steps involved in using a machine learning device to convert far-infrared images into visible light images and then using the visible light images as training data to perform machine learning on an object detection model.

[0024] Figure 10 This is to explain the passage. Figure 8 The flowchart illustrates the steps of a far-infrared imaging device in converting far-infrared images into visible light images and detecting objects from those images. Detailed Implementation

[0025] (Implementation Method 1)

[0026] Figure 1 This is a structural diagram of the machine learning device 200 according to Embodiment 1. The machine learning device 200 includes a training data storage unit 10, a far-infrared image training data acquisition unit 20, a visible light image training data acquisition unit 30, a visible light image generation model learning unit 40, and a learned model storage unit 50.

[0027] Visible light has a wavelength range of 380–780 nm, while far-infrared light has a wavelength range of 8–15 μm. Visible light allows humans to clearly identify objects such as people, animals, and license plates. However, visible light cameras have the following problems: they are weak in backlighting, weak in fog or haze, and cannot photograph objects at night without a light source.

[0028] On the other hand, far-infrared cameras are resistant to fog or haze, and can capture blurry images of objects even at night or in places without light sources. However, far-infrared cameras suffer from problems such as difficulty in accurately identifying objects, low resolution, and significant distortion because they cannot capture objects in detail.

[0029] Here, images captured by a far-infrared imaging device are designated as far-infrared images, and images captured by a visible light imaging device are designated as visible light images. It is known that far-infrared images and visible light images are correlated.

[0030] The training data storage unit 10 stores training data of a large number of far-infrared images for all time periods and training data of a large number of visible light images for all time periods.

[0031] The far-infrared image training data acquisition unit 20 acquires far-infrared images for a first predetermined time period from the training data storage unit 10 as training data and provides them to the visible light image generation model learning unit 40. The visible light image training data acquisition unit 30 acquires visible light images for a second predetermined time period from the training data storage unit 10 as training data and provides them to the visible light image generation model learning unit 40.

[0032] Because the far-infrared images from the first specified time period and the visible light images from the second specified time period are taken at different times, the composition of the images, i.e., the arrangement of the people or objects in the photos, usually differs. However, since the arrangement of objects remains constant throughout the day, there are also combinations where the far-infrared images from the first specified time period and the visible light images from the second specified time period have the same image composition. As an example, the first specified time period is nighttime, and the second specified time period is daytime.

[0033] The visible light image generation model learning unit 40 generates a learned visible light image generation model that generates visible light images for a second specified time period based on far-infrared images from a first specified time period. Specifically, the far-infrared images from the first specified time period and the visible light images from the second specified time period are used as training data. Machine learning is performed on the visible light image generation network and the far-infrared image recognition network in a Generative Adversarial Network (GAN). The visible light image generation network generates a visible light image when a far-infrared image is input, and the far-infrared image recognition network identifies whether the input far-infrared image from the first specified time period is genuine or counterfeit.

[0034] Based on the learning results of GAN, a learned model of the visible light image generation network, namely generation model 60, and a learned model of the far-infrared image recognition network, namely recognition model 70, are generated. Generation model 60 is a learned model of the visible light image generation network that generates visible light images for a second specified time period based on far-infrared images from a first specified time period. Recognition model 70 is a learned model of the far-infrared image recognition network that identifies whether a far-infrared image was captured in the first specified time period. The visible light image generation model learning unit 40 stores generation model 60 and recognition model 70 in the learned model storage unit 50.

[0035] Here, we will explain GANs. GANs are a type of machine learning. GANs are machine learning methods that enable the generator network and the recognition network to learn adversarially and improve each other's performance. As a result of the learning, we can obtain learned models of both the generator network and the recognition network.

[0036] Generative networks are neural networks that generate new images based on learned image features when given input signals such as random noise.

[0037] A recognition network is a neural network that, when given an input image, identifies whether the input image is a genuine image (in other words, a target image) or a forged image (in other words, an image generated by a generative network).

[0038] As learning methods for GANs, Pix2Pix and Cycle GAN are explained respectively.

[0039] Pix2Pix is ​​a Generative Adversarial Network (GAN) that enables the generator network and the recognition network to perform adversarial learning separately. The generator network generates image data when input image data is received, and the recognition network identifies whether the input image data is the target image data (i.e., genuine product) or the generated image data (i.e., counterfeit product). In Pix2Pix, the learning result is a set of learned models for both the generator and recognition networks.

[0040] In Pix2Pix, the input image data contained in group X and the target image data contained in group Y are learned as a one-to-one pair, so that the generative network learns the features (transformation characteristics) from group X to group Y.

[0041] In Pix2Pix, the paired input and target image data need to have consistent contour information. This is because Pix2Pix predicts the texture information within the contours.

[0042] When input image data of group X is fed into a learned model of a generative network that has learned the transformation properties, the generative network generates generated image data with features of group Y.

[0043] Here, the generator network is learned to enable the recognition network to identify the generated image data as genuine, while the recognition network is learned adversarially to identify the generated image data as forgeries.

[0044] Cycle GAN is a type of GAN where the generator network and the recognition network engage in adversarial learning, one in each network. The generator network generates target image data when input image data is received, while the recognition network identifies whether the input image data is the target image data (i.e., the genuine product) or the generated image data (i.e., the counterfeit product). In Cycle GAN, the learning result is two sets of learned models for the generator and recognition networks.

[0045] In Cycle GAN, the input image data contained in group X and the target image data contained in group Y are learned as a pair, so that the first generator network learns the features (transformation characteristics) from group X to group Y. And the input image data contained in group Y and the target image data contained in group X are learned as a pair, so that the second generator network learns the features (transformation characteristics) from group Y to group X.

[0046] Furthermore, the input image data of group X is input into a first generator network that has learned the transformation characteristics from group X to group Y, generating generated image data with the characteristics of group Y. This generated image data is then input into a second generator network that has learned the transformation characteristics from group Y to group X, and the network is trained to generate generated image data that is then used to input image data for group X. In parallel, the input image data of group Y is input into the second generator network that has learned the transformation characteristics from group Y to group X, generating generated image data with the characteristics of group X. This generated image data is then input into the first generator network that has learned the transformation characteristics from group X to group Y, and the network is trained to generate generated image data that is then used to input image data for group Y. In other words, Cycle GAN learns by considering the interconversion between group X and group Y; the higher the correlation between group X and group Y, the more effectively the transformation can be performed.

[0047] Here, the first recognition network is fed with generated image data generated by the first generation network and image data of group Y, and learns to identify whether the image input to the first recognition network is genuine (i.e., image data of group Y) or counterfeit (i.e., generated image data generated by the first generation network).

[0048] In addition, the second recognition network is fed with the generated image data generated by the second generation network and the image data of group X, so that it can learn to identify whether the image input to the second recognition network is genuine (i.e., the image data of group X) or counterfeit (i.e., the generated image data generated by the second generation network).

[0049] Here, the learning process is performed so that the first recognition network can identify the generated image data generated by the first generation network as genuine.

[0050] In addition, the second generator network learns to recognize that the generated image data generated by the second recognition network is genuine.

[0051] In Cycle GAN, the input image data and the target image data are a pair, and the contour information does not need to be consistent. This is because Cycle GAN predicts the texture information inside the contour while predicting the contour.

[0052] When input image data of group X is fed into the learned model of the first generative network that has learned the transformation characteristics, the generative network generates generated image data with the features of group Y.

[0053] When input image data of group Y is fed into the learned model of the second generative network that has learned the transformation characteristics, the generative network generates generated image data with the features of group X.

[0054] As mentioned above, Pix2Pix and Cycle GAN can be used for generative network learning to generate images with features of group Y from images of group X.

[0055] For example, if the image in group X is an edge image of a bag and the image in group Y is a photographic image of a bag, a generative network can be made to learn to output a photographic image when given an edge image as input.

[0056] In Implementation 1, Cycle GAN is used as the GAN. In the above description of Cycle GAN, group X is the far-infrared image, and group Y is the visible light image.

[0057] Figure 2 This is a structural diagram of the far-infrared imaging device 300 according to Embodiment 1. The far-infrared imaging device 300 includes a learned model storage unit 50, an image processing unit 100, and an image display unit 110. The image processing unit 100 includes a far-infrared image acquisition unit 80, a far-infrared image recognition unit 85, and a visible light image generation unit 90. The learned model storage unit 50 and... Figure 1 The learned model storage unit 50 has the same structure as the learned model storage unit 50, and stores the learned model of the visible light image generation network (i.e., generation model 60) and the learned model of the far-infrared image recognition network (i.e., recognition model 70) generated by the visible light image generation model learning unit 40.

[0058] The far-infrared image acquisition unit 80 acquires far-infrared images captured by the far-infrared imaging device 300 and provides them to the far-infrared image recognition unit 85. The far-infrared image recognition unit 85 inputs the far-infrared image acquired by the far-infrared image acquisition unit 80 into the recognition model 70 read from the learned model storage unit 50, and identifies whether the acquired far-infrared image was captured within a first predetermined time period. If the acquired far-infrared image was captured within the first predetermined time period, the far-infrared image recognition unit 85 provides it to the visible light image generation unit 90; if the acquired far-infrared image was not captured within the first predetermined time period, it does not provide it to the visible light image generation unit 90.

[0059] The visible light image generation unit 90 inputs the far-infrared image provided by the far-infrared image recognition unit 85 into the generation model 60 read from the learned model storage unit 50, and generates a visible light image. Thus, the visible light image generation unit 90 generates a visible light image if the far-infrared image recognition unit 85 determines that the far-infrared image was captured within a first predetermined time period; otherwise, it does not generate a visible light image if the far-infrared image was not captured within the first predetermined time period.

[0060] Here, as described above, the generative model 60 is a learned model of a visible light image generation network that has undergone machine learning in an adversarial generative network using far-infrared images of the first predetermined time period and visible light images of the second predetermined time period as training data, and generates visible light images of the second predetermined time period from far-infrared images of the first predetermined time period.

[0061] The image display unit 110 displays a visible light image generated by the visible light image generation unit 90. The image display unit 110 may also display a far-infrared image acquired by the far-infrared image acquisition unit 80. The image display unit 110 may also display either a far-infrared image or a visible light image, or both a far-infrared image and a visible light image, or a composite display of a far-infrared image and a visible light image.

[0062] As described above, in situations where visible light cameras cannot capture objects due to backlighting, fog, haze, or nighttime, a far-infrared camera capable of capturing blurry images of the object can be used to capture the approximate portion of the object. A visible light image generation network, learned from a far-infrared image containing the blurry object using a GAN, can then be used to generate detailed images of the object. In this way, visible light images that clearly or in detail identify the object can also be obtained from far-infrared images.

[0063] More specifically, a generative model 60 is generated, which is a learned model of a visible light image generation network that generates visible light images when a far-infrared image is input. By inputting far-infrared images of objects during periods when it is impossible to capture them clearly or in detail using a visible light camera (e.g., nighttime), visible light images that allow for clear or detailed identification of the objects can be obtained even during these periods.

[0064] Additionally, a recognition model 70 is generated. This recognition model 70 is a learned model of a far-infrared image recognition network that identifies whether an input far-infrared image is a far-infrared image within a first predetermined time period. A high-precision far-infrared image identified as a far-infrared image within the first predetermined time period by the recognition model 70 is provided to the learned model of the visible light image generation network, namely, the generation model 60. As long as the probability density distribution of the feature quantities of a far-infrared image is similar to that of the learned far-infrared image, the visible light image generation network can convert it into a visible light image with high precision.

[0065] Figure 3 (a) and Figure 3 (b) is a diagram illustrating the defined time periods. The first defined time period is the nighttime period, and the second defined time period is the daytime period. For example... Figure 3As shown in (a), the daytime period can be from 9:00 to 15:00, and the nighttime period can be from 21:00 to 3:00 the next day. Figure 3 As shown in (b), the daytime period can be from 6:00 to 18:00, and the nighttime period can be from 18:00 to 6:00 the next day.

[0066] Visible light cameras can clearly capture images of people and other objects during the day, but they cannot capture such images clearly at night. On the other hand, far-infrared cameras struggle to capture the detailed textures of people and other objects even at night, but they can clearly capture the shape of the objects.

[0067] Therefore, the first specified time period is the time period during which the object can be clearly photographed with a visible light camera, and the second specified time period is the time period during which the object cannot be clearly photographed with a visible light camera.

[0068] Furthermore, regarding visible light images captured by a visible light camera, not only the time period but also the season, weather, and location are specified, thereby improving the quality of the visible light images used as training data. For example, consider visible light images captured on a sunny day in March, or visible light images captured at a specific floor level. Additionally, for visible light images captured by a visible light camera, it is preferable to select conditions such as time period, season, weather, and location to ensure that the amount of visible light is within a specified range.

[0069] On the other hand, for far-infrared images captured by a far-infrared camera, not only the time period is limited, but also the season, weather, and location are specified, thereby improving the quality of the far-infrared images used as training data. Furthermore, regarding far-infrared images captured by a far-infrared camera, it is preferable to select conditions such as time period, season, weather, and location so that the temperature is within a specified range.

[0070] In this way, by using visible light and far-infrared images with defined time periods, seasons, weather, and locations for learning in GAN, it is possible to generate high-precision learned models for visible light image generation networks and far-infrared image recognition networks.

[0071] Figure 4 This is to explain the passage. Figure 1 The flowchart illustrates the steps of the machine learning device 200 performing machine learning on the generation model 60 of the visible light image generation network and the recognition model 70 of the far-infrared image recognition network. Here, as an example, the first predetermined time period is defined as nighttime and the second predetermined time period as daytime.

[0072] The far-infrared image training data acquisition unit 20 acquires a nighttime far-infrared image from the training data storage unit 10 (S10). The visible light image training data acquisition unit 30 acquires a daytime visible light image from the training data storage unit 10 (S20).

[0073] The visible light image generation model learning unit 40 performs machine learning in the GAN by using nighttime far-infrared images and daytime visible light images as training data, and generates a generation model 60 (S30) that generates daytime visible light images based on nighttime far-infrared images.

[0074] The visible light image generation model learning unit 40 performs machine learning in the GAN by using nighttime far-infrared images and daytime visible light images as training data to generate a recognition model 70 (S40) that identifies whether a far-infrared image is a far-infrared image taken at night. Here, the illustration shows S40 being executed after S30, but in a typical GAN, S30 and S40 are executed repeatedly.

[0075] Figure 5 This is to explain the passage. Figure 2 The flowchart illustrates the steps of converting a far-infrared image into a visible light image using a generation model 60 of a visible light image generation network and a recognition model 70 of a far-infrared image recognition network. Here, as an example, a first predetermined time period is defined as nighttime and a second predetermined time period as daytime.

[0076] Input the far-infrared image captured by the far-infrared imaging device 300 (S50).

[0077] The recognition model 70, which uses machine learning in GAN, identifies whether the input far-infrared image is a far-infrared image taken at night (S60).

[0078] If the input far-infrared image is a far-infrared image taken at night (S70 "Yes"), proceed to step S80; if the input far-infrared image is not a far-infrared image taken at night (S70 "No"), end.

[0079] A generative model 60, which uses nighttime far-infrared images and daytime visible light images as training data, performs machine learning in a GAN to convert the input far-infrared image into a daytime visible light image (S80).

[0080] Output the generated daytime visible light image (S90). The output daytime visible light image is used for object detection, etc.

[0081] According to Implementation Method 1, nighttime far-infrared images and daytime visible light images are used as training data. A visible light image generation network that generates visible light images based on far-infrared images is trained in a GAN. The far-infrared images are converted into visible light images through the generation model of the visible light image generation network trained in the GAN, thereby obtaining clear images that are independent of day and night.

[0082] By incorporating the generative model 60 into the far-infrared imaging device 300, the far-infrared imaging device 300 can always identify people and animals like visible light images, regardless of the time period. The generative model is a learned model of a visible light image generation network that generates visible light images if a far-infrared image is input.

[0083] In addition, by mounting the recognition model 70 on the far-infrared imaging device 300, the generation accuracy of the visible light image based on the generation model 60 can be improved. The recognition model 70 is a learned model of the far-infrared image recognition network that identifies whether the input far-infrared image is a far-infrared image of a first predetermined time period.

[0084] (Variation Example 1)

[0085] Compared to the machine learning device 200 in Embodiment 1, the training data and the structure and operation of the visible light image generation model learning unit 40 are different; therefore, only the differences will be described. In Modification 1, the training data for the visible light images of the second predetermined time period are images with the same composition obtained by photographing the same object in the same location as the training data for the far-infrared images of the first predetermined time period, and the far-infrared images of the first predetermined time period and the visible light images of the second predetermined time period are paired. Therefore, the visible light image generation model learning unit 40 uses Pix2Pix as the learning method for GAN.

[0086] The visible light image generation model learning unit 40 performs machine learning on Pix2Pix in pairs using training data of far-infrared images over a first predetermined time period and training data of visible light images over a second predetermined time period, generating a learned model, i.e., generation model 60, of the visible light image generation network that generates visible light images when input far-infrared images. In the above description of Pix2Pix, group X is far-infrared images, and group Y is visible light images.

[0087] Furthermore, the visible light image generation model learning unit 40 generates a recognition model 70, which is a learned model of a far-infrared image recognition network that identifies whether the input far-infrared image is a far-infrared image from a first predetermined time period. In the above description of Pix2Pix, group Y is a far-infrared image, and group X is a visible light image.

[0088] If a large number of images can be prepared by taking pictures of the same objects in the same location with the same composition, the generation accuracy of the visible light images generated by the generation model 60 of the visible light image generation network, which generates visible light images from input far-infrared images, can be improved.

[0089] (Variation Example 2)

[0090] Compared to the machine learning device 200 in Embodiment 1, the training data and the structure and operation of the visible light image generation model learning unit 40 are different, so only the differences will be described. In Modification 2, the training data for the visible light images are visible light images taken at the same location as the training data for the far-infrared images of the first predetermined time period, illuminating the same object in the same way and with the same composition during the same time period.

[0091] The visible light image generation model learning unit 40 performs machine learning on the GAN in pairs with the training data of the far-infrared image during the first specified time period and the training data of the visible light image projected during the first specified time period, to generate a learned model of the visible light image generation network that generates a visible light image when the input is a far-infrared image, namely the generation model 60.

[0092] Here, as a GAN, Pix2Pix can also be used. When using Pix2Pix, it is desirable that the training data for far-infrared images and visible light images are acquired as simultaneously as possible. This can be achieved by acquiring training data for far-infrared images just before the light is projected and training data for visible light images immediately after the light is projected. In the above description of Pix2Pix, group X is the far-infrared image, and group Y is the visible light image.

[0093] If a large number of images can be prepared that capture the same object at roughly the same time and in the same location with the same composition, the generation accuracy of the visible light images generated by the generation model 60 of the visible light image generation network, which generates visible light images when input far-infrared images, can be improved.

[0094] (Implementation Method 2)

[0095] In Embodiment 2, at least one of the first and second predetermined time periods is divided into multiple time periods. The visible light image generation model learning unit 40 learns the generation model 60 and the recognition model 70 according to each segmented time period. The far-infrared image recognition unit 85 and the visible light image generation unit 90 switch between using the generation model 60 and the recognition model 70 according to the time period.

[0096] Figure 6 (a) and Figure 6(b) is a diagram illustrating the segmented time periods. The first segment is the nighttime period, and the second segment is the daytime period. For example... Figure 6 As shown in (a), the nighttime period can also be divided into two time periods with different characteristics of the far-infrared images. By switching the recognition model 70 to match the shooting time period of the far-infrared images acquired by the far-infrared image recognition unit 85, the accuracy of recognition can be improved.

[0097] like Figure 6 As shown in (b), the daytime period can also be divided into three time periods with different characteristics of visible light images, and the nighttime period can be divided into two time periods with different characteristics of far-infrared images. The far-infrared image recognition unit 85 and the visible light image generation unit 90 switch between recognition model 70 and generation model 60 that correspond to the shooting time period, thereby improving the accuracy of recognition and generation, and enabling the diversification of the generated visible light images.

[0098] In addition, the time period is the same as in Implementation 1, but the accuracy of the far-infrared image recognition unit 85 and the visible light image generation unit 90 can be improved by preparing the recognition model 70 and the generation model 60 according to the season, weather and location.

[0099] (Implementation Method 3)

[0100] The visible light image training data acquisition unit 30 of the machine learning device 200 in Embodiment 3 has a different structure and operation than that of the machine learning device 200 in Embodiment 1, except that its structure and operation are the same. Descriptions of structures and operations common to Embodiment 1 are omitted where appropriate.

[0101] Visible light cameras can clearly capture images of people and license plates during a clear day, but from the perspective of personal information protection, capturing images of people and license plates that are not anonymous presents a problem. Therefore, in the machine learning device 200 of Implementation 3, visible light images that have been anonymized in a way that makes it impossible to identify the specified objects are used as training data.

[0102] The visible light image training data acquisition unit 30 of the machine learning device 200 performs concealment processing on the visible light image acquired from the training data storage unit 10 in a way that makes the specified object unrecognizable, and provides the training data of the visible light image that has been concealed in a way that makes the specified object unrecognizable to the visible light image generation model learning unit 40.

[0103] Similar to Embodiment 1, the visible light image generation model learning unit 40 generates a learned visible light image generation model that generates a visible light image for a second predetermined time period based on a far-infrared image from a first predetermined time period. However, the visible light images used as training data in the learning process are anonymized in a way that prevents the identification of the specified objects. Therefore, as a result of the learning, the generation model 60 of the generated visible light image generation network converts the far-infrared image into a visible light image with the specified objects anonymized. This results in easily viewable images that protect personal information and portrait rights, regardless of whether it is nighttime or daytime.

[0104] When the visible light image generation unit 90 in the image processing apparatus 100 generates a visible light image based on the far-infrared image using the generation model 60, it becomes a visible light image that has been concealed in a way that makes the specified object unrecognizable. Therefore, it is not necessary to conceal the specified object in the generated visible light image.

[0105] The specified object is the object of personal information protection or portrait rights protection. For example, it can be a head containing a face, or a license plate of a vehicle or motorcycle. Here, the specified object is set as a head containing a face, or a license plate of a vehicle or motorcycle, but it can be any object of personal information protection or portrait rights protection. In order to make the protection of rights more effective, the specified object can also be the whole body of a person, or the entire vehicle or motorcycle.

[0106] The pixel values ​​within a specified object area are replaced with specified pixel values ​​(e.g., black) to achieve an occlusion process, making the specified object unrecognizable. Here, the specified pixel values ​​replace the pixel values ​​of the specified object, but as long as the specified object is unrecognizable, it is acceptable; for example, a hypothetical person, car, etc., depicted by computer graphics (CG) could be used for replacement. Alternatively, the specified pixel values ​​can be changed to red, blue, green, etc., for each image.

[0107] Furthermore, the determination of whether a specified object can be identified can be made, for example, using a learned object recognition model that has been learned in a way that enables the identification of the specified object. Alternatively, as another method, a person can also determine whether the specified object can be identified by visual inspection.

[0108] (Variation Example 1)

[0109] As another example of concealment processing to make the specified objects unrecognizable, concealment processing can also be performed according to the specified size of each specified object, which cannot be accurately identified, so that only visible light images of all specified objects that are below the specified size defined for each specified object are used as training data.

[0110] (Variation Example 2)

[0111] As another example of concealing a specified object in a way that makes it unrecognizable, if the specified object is larger than the specified size defined for each specified object, the specified object can be replaced with a specified pixel value (e.g., black) to conceal it, making the specified object unrecognizable.

[0112] In Implementation 3 and its variations, a learned model of a visible light image generation network is generated. This network uses far-infrared images and visible light images obscured in a way that makes the specified object unrecognizable as training data. Machine learning is then performed in a GAN to generate visible light images that are obscured in a way that makes the specified object unrecognizable when the input far-infrared image is used. Thus, it is possible to generate visible light images that are easily viewable except for the specified object, while complying with personal information protection or portrait rights protection.

[0113] In addition, the visible light images generated by GANs are pseudo-images generated based on the features of the learned images, thus making it easy to anonymize specified objects.

[0114] Furthermore, by learning the model of the visible light image generation network in advance, it is not necessary to perform hidden processing on the specified objects when using it.

[0115] (Implementation Method 4)

[0116] Far-infrared cameras are resistant to fog or haze, and can blurryly photograph people and animals even in places without light sources, such as at night. However, far-infrared cameras cannot capture details of objects such as people or animals and lack color information, resulting in low accuracy in object detection or recognition. Therefore, in Implementation 4, a visible light image generation network is learned in a GAN to generate visible light images from far-infrared images. Object detection is then performed on the visible light images generated from the far-infrared images using the generation model of the visible light image generation network, thereby improving the accuracy of object detection.

[0117] Figure 7 This is a structural diagram of the machine learning apparatus 400 according to Embodiment 4. The machine learning apparatus 400 includes an image processing apparatus 100, a training data storage unit 120, an object detection model learning unit 130, and an object detection model storage unit 140. The image processing apparatus 100 includes a far-infrared image acquisition unit 80, a visible light image generation unit 90, and a visible light image recognition unit 95. Although in Figure 7 Not shown in the diagram, but included in this embodiment is different from the machine learning device 400. Figure 1 Machine learning device 200.

[0118] In implementation method 4, Figure 1 The learning unit 40 of the visible light image generation model uses far-infrared images of a first predetermined time period and visible light images of a second predetermined time period as training data. It performs machine learning in GAN for the visible light image generation network that generates visible light images when inputting far-infrared images, and the visible light image recognition network that identifies whether the input visible light image as a visible light image of the second predetermined time period is genuine or counterfeit when inputting visible light images.

[0119] As a result of GAN-based learning, a generative model 60, which is a learned model of a visible light image generation network, and a recognition model 75, which is a learned model of a visible light image recognition network, are generated. The generative model 60 is a learned model of a visible light image generation network that generates a visible light image for a second predetermined time period based on a far-infrared image taken during a first predetermined time period. The recognition model 75 is a learned model of a visible light image recognition network that identifies whether an item is genuine or counterfeit based on a visible light image captured during the first predetermined time period. The visible light image generation model learning unit 40 stores the generative model 60 and the recognition model 75 in the learned model storage unit 50. The visible light image generation model learning unit 40 provides the generative model 60 to the visible light image generation unit 90. Furthermore, the visible light image generation model learning unit 40 provides the recognition model 75 to the visible light image recognition unit 95.

[0120] The far-infrared image acquisition unit 80 acquires a far-infrared image and provides it to the visible light image generation unit 90.

[0121] The visible light image generation unit 90 inputs the far-infrared image provided by the far-infrared image acquisition unit 80 into the learned model, i.e. the generation model, of the visible light image generation network to generate a visible light image, and then provides the generated visible light image to the visible light image recognition unit 95.

[0122] The visible light image recognition unit 95 inputs the visible light image provided by the visible light image generation unit 90 into the learned model, i.e., the recognition model, of the visible light image recognition network. It identifies whether the input visible light image is genuine or counterfeit. Visible light images identified as genuine are saved to the training data storage unit 120, while those identified as counterfeit are not. Here, if the input visible light image is identified as genuine, it is determined to be normal; if it is identified as counterfeit, it is determined to be abnormal.

[0123] The training data storage unit 120, for the input visible light image, assigns labels (e.g., object names) and the position and size of bounding boxes for one or more objects within the visible light image, accumulating this data as training data. For example, when 1000 objects have been accumulated, the training data generation is complete, and the object detection model learning unit 130 is instructed to begin learning. Here, as an example of assigning labels to objects, there are faces, pedestrians, cars, bicycles, etc.

[0124] The object detection model learning unit 130 accumulates sufficient training data in the training data storage unit 120. When it receives an instruction to start learning, it uses the training data accumulated in the training data storage unit 120 to perform machine learning on the object detection model. Here, as an example, the object detection model utilizes YOLO (You Only Memory Look Once), which utilizes a convolutional neural network (CNN) as a representative model for object detection. The object detection model learning unit 130 saves the machine-learned object detection model in the object detection model storage unit 140.

[0125] Figure 8 This is a structural diagram of the far-infrared imaging device 500 according to Embodiment 4. The far-infrared imaging device 500 includes an image processing unit 100, an object detection model storage unit 140, an object detection unit 150, and an image display unit 160. The image processing unit 100 and... Figure 7 The image processing unit 100 of the machine learning device 400 also includes a far-infrared image acquisition unit 80, a visible light image generation unit 90, and a visible light image recognition unit 95. The object detection model storage unit 140 and... Figure 7 The structure is the same as that of the object detection model storage unit 140, which stores the object detection model learned by the object detection model learning unit 130.

[0126] The far-infrared image acquisition unit 80 acquires the far-infrared image captured by the far-infrared imaging device 500 and provides it to the visible light image generation unit 90.

[0127] The visible light image generation unit 90 inputs the far-infrared image provided by the far-infrared image acquisition unit 80 into the learned model, i.e. the generation model, of the visible light image generation network to generate a visible light image, and then provides the generated visible light image to the visible light image recognition unit 95.

[0128] The visible light image recognition unit 95 inputs the visible light image provided by the visible light image generation unit 90 into the learned model, i.e., the recognition model, of the visible light image recognition network, and identifies whether the input visible light image is genuine or counterfeit. The visible light image identified as genuine is provided to the object detection unit 150, while the visible light image identified as counterfeit is not provided to the object detection unit 150.

[0129] The object detection unit 150 detects objects within the visible light image by inputting the input visible light image into the object detection model stored in the object detection model storage unit 140. The input visible light image and the detected object information are provided to the image display unit 160. The image display unit 160 combines the visible light image, labels, and bounding boxes and displays them on the screen.

[0130] As described above, in situations where visible light cameras cannot capture objects due to backlighting, fog, haze, or nighttime, far-infrared cameras capable of capturing blurry images of objects can be used to capture the object's outline. Detailed images of the object can then be generated using a visible light image generation network learned in a GAN from the far-infrared image containing the blurred image. Therefore, object recognition and detection can be performed with high accuracy even for far-infrared images.

[0131] Figure 9 This is to explain the passage. Figure 7 The flowchart illustrates the steps of the machine learning device 400 in converting far-infrared images into visible light images and using the visible light images as training data to perform machine learning on an object detection model. Here, as an example, the first defined time period is set to nighttime and the second defined time period to daytime.

[0132] The far-infrared image acquisition unit 80 inputs a far-infrared image (S100).

[0133] The visible light image generation unit 90 uses a generation model of a visible light image generation network that uses nighttime far-infrared images and daytime visible light images as training data and performs machine learning in a GAN to convert the input far-infrared image into a daytime visible light image (S110).

[0134] The visible light image recognition unit 95 uses a recognition model of a visible light image recognition network that has undergone machine learning in GAN to determine whether the converted daytime visible light image is normal (S120). If the converted daytime visible light image is normal (S130 "Yes"), proceed to step S140; if the converted daytime visible light image is abnormal (S130 "No"), return to step S100.

[0135] The visible light image recognition unit 95 accumulates the daytime visible light images that are determined to be normal as training data in the training data storage unit 120 (S140).

[0136] The object detection model learning unit 130 uses daytime visible light images accumulated as training data to perform machine learning on the object detection model (S150).

[0137] Figure 10 This is to explain the passage. Figure 8 The flowchart shows the steps of the far-infrared imaging device 500 in converting far-infrared images into visible light images and detecting objects from the visible light images.

[0138] The far-infrared image acquisition unit 80 inputs the far-infrared image captured by the far-infrared imaging device 500 (S200).

[0139] The visible light image generation unit 90 uses a generation model of a visible light image generation network that uses nighttime far-infrared images and daytime visible light images as training data and performs machine learning in a GAN to convert the input far-infrared image into a daytime visible light image (S210).

[0140] The visible light image recognition unit 95 uses a recognition model of a visible light image recognition network that has undergone machine learning in GAN to determine whether the converted daytime visible light image is normal (S220). If the converted daytime visible light image is normal (S230 "Yes"), proceed to step S240; if the converted daytime visible light image is abnormal (S230 "No"), return to step S200.

[0141] The object detection unit 150 uses an object detection model to detect objects from a daytime visible light image that is determined to be normal (S240). The image display unit 160 displays the objects detected in the daytime visible light image (S250).

[0142] (Variation Example 1)

[0143] In embodiment 4, a visible light image recognition unit 95 is provided in the image processing apparatus 100, but the visible light image recognition unit 95 may not be provided. In addition, the visible light image recognition unit 95 may be provided only in the image processing apparatus 100 of the machine learning apparatus 400, or it may be provided only in the image processing apparatus 100 of the far-infrared imaging apparatus 500.

[0144] (Variation Example 2)

[0145] exist Figure 8In the far-infrared imaging device 500, the object detection unit 150 performs object detection only based on the visible light image identified as a genuine item by the visible light image recognition unit 95. However, the object detection unit 150 can also perform object detection based on the visible light image identified as a counterfeit item by the visible light image recognition unit 95. In this case, the reliability threshold B for object detection when the visible light image recognition unit 95 identifies the item as a counterfeit item is higher than the reliability threshold A for object detection when the visible light image recognition unit 95 identifies the item as a genuine item. That is, if the reliability of object detection is not higher than the threshold B (>A) when the visible light image recognition unit 95 identifies the item as a counterfeit item, it is not considered as an object being detected, thereby improving the fit rate and the reproduction rate.

[0146] Furthermore, when a high reproduction rate is required, the difference between the reliability threshold B of object detection when the visible light image recognition unit 95 identifies the object as a counterfeit and the reliability threshold A of object detection when the visible light image recognition unit 95 identifies the object as a genuine object can be reduced. When a high fit rate is required, the difference between the reliability threshold B of object detection when the visible light image recognition unit 95 identifies the object as a counterfeit and the reliability threshold A of object detection when the visible light image recognition unit 95 identifies the object as a genuine object can be increased.

[0147] Furthermore, when the visible light image recognition unit 95 outputs the probability that the object is genuine, it is possible that a higher probability of being genuine lowers the threshold for object detection reliability, and a lower probability of being genuine raises the threshold for object detection reliability. Moreover, the reliability of object detection is an evaluation value based on the probability that an object is detected within a certain area.

[0148] According to embodiment 4, object recognition or detection is performed after generating a visible light image from a far-infrared image from which the object is difficult to accurately identify, thereby enabling high-precision object recognition or detection even for far-infrared images.

[0149] Furthermore, by learning an object detection network based on the recognition results of the already learned model of the recognition network, the accuracy of the object detection network can be improved.

[0150] Furthermore, by inferring from the recognition results of the learned model of the recognition network, the accuracy of the object detection network can be improved. Additionally, by adjusting the reliability threshold of object detection based on the recognition probability of the learned model of the recognition network, it is easy to handle a variety of applications.

[0151] In addition, it does not require a visible light camera; it can perform object identification and detection with high precision using only a far-infrared camera, making it suitable for miniaturization and weight reduction of devices.

[0152] The various processing operations described above by the image processing device 100, machine learning device 200, far-infrared imaging device 300, machine learning device 400, and far-infrared imaging device 500 can, of course, be implemented using hardware such as a CPU and memory, or through software such as firmware stored in ROM (read-only memory) or flash memory, or computer software. The firmware or software program can be recorded on a readable recording medium such as a computer and provided, or it can be transmitted and received from a server via wired or wireless networks, or it can be transmitted and received as data broadcast via terrestrial waves or satellite digital broadcasting.

[0153] The present invention has been described above based on embodiments. Those skilled in the art will understand that the embodiments are illustrative, and the combinations of the constituent elements and processing procedures can have various modifications, and such modifications are also within the scope of the present invention.

[0154] Industrial applicability

[0155] This invention enables the use of technology to convert far-infrared images into visible light images.

[0156] Symbol Explanation

[0157] 10 Training Data Storage Department

[0158] 20 Far-infrared Image Training Data Acquisition Department

[0159] 30 Visible Light Image Training Data Acquisition Department

[0160] 40 Visible Light Image Generation Model Learning Unit

[0161] 50 Learned Model Storage Department

[0162] 60 Generative Models

[0163] 70 recognition model

[0164] 80 Far-infrared Image Acquisition Unit

[0165] 85 Far-infrared Image Recognition Unit

[0166] 90 Visible light image generation unit

[0167] 95 Visible Light Image Recognition Unit

[0168] 100 Image Processing Device

[0169] 110 Image Display Unit

[0170] 120 Training Data Storage Department

[0171] 130 Object Detection Model Learning Department

[0172] 140 Object Detection Model Storage Unit

[0173] 150 Object Detection Department

[0174] 160 Image Display Unit

[0175] 200 machine learning devices

[0176] 300 far-infrared imaging device

[0177] 400 machine learning devices

[0178] 500 far-infrared imaging device.

Claims

1. A machine learning device, characterized in that, include: The far-infrared image training data acquisition unit acquires far-infrared images for a first predetermined time period; The visible light image training data acquisition unit acquires visible light images over a second predetermined time period; as well as The visible light image generation model learning unit uses far-infrared images from the first specified time period and visible light images from the second specified time period as training data to perform machine learning in a generative adversarial network to generate a learned generative model. This learned generative model is used to generate visible light images from the second specified time period based on the far-infrared images from the first specified time period. The visible light image training data acquisition unit cannot accurately identify the specified size of each specified object according to the definition of each specified object. If all specified objects in the visible light image are below the specified size, the visible light image is input into the visible light image generation model learning unit.

2. The machine learning apparatus as described in claim 1, characterized in that, The first specified time period is the period when the temperature is within a first specified range, and the second specified time period is the period when the light intensity is within a second specified range.

3. The machine learning apparatus as described in claim 1 or 2, characterized in that, The visible light image generation model learning unit also generates a learned recognition model by performing machine learning in a generative adversarial network. The learned recognition model is used to identify whether the far-infrared image is a far-infrared image captured in the first specified time period.

Images