Infrared information display device, infrared information display method, learning data generation device, and learning system

Abstract

The present disclosure relates to an infrared information display device provided with: a first blackbody furnace; a second blackbody furnace having a temperature different from that of the first blackbody furnace; a drive control unit which, on the basis of temperature information about a far-infrared image that has been input, performs a drive switching control to switch radiation light to be reflected by a reflecting mirror from radiation light of the first blackbody furnace to radiation light of the second blackbody furnace, whereby a temperature corresponding to the temperature information is reflected; and a reflecting unit including a plurality of two-dimensionally arranged reflecting mirrors which respectively correspond to pixels of the far-infrared image and of which orientations are changed by the drive switching control.

Description

[Technical field]

[0001] The present disclosure relates to an infrared information display device, an infrared information display method, a learning data generation device, and a learning system. [Background technology]

[0002] A technology for constructing a high-quality image from a low-quality image captured by a camera has been realized (for example, Non-Patent Document 1). For example, a method for generating a high-quality image from a low-resolution image related to a visible light image using CNN (Convolutional Neural Networks) has been disclosed. In such a method, CG (Computer Graphics) is displayed on a display, photographed by a low-resolution camera, and machine learning is performed using a pair of the photographed image and the original CG as learning data to construct a network that reproduces a high-resolution image.

[0003] It is desirable to apply such learning on visible light images to learning on infrared images. However, in the case of a far-infrared camera, a display or the like is required to convert the brightness information of an image into temperature information and present it, so it is not easy to apply a method similar to that for visible light to a far-infrared camera. For example, a display technology has been disclosed that presents two-dimensional temperature information by arranging resistors two-dimensionally, passing a current through each of them, and controlling the amount of heat generated by each resistor by current control (for example, Non-Patent Document 2). However, such a method requires time for the temperature of the resistors to stabilize. Therefore, it is not suitable for generating a large amount of learning data, and it is also necessary to correct temperature errors due to differences in characteristics between resistors.

[0004] For example, in Non-Patent Document 1, a high-resolution image is estimated from a low-resolution visible light image, but this low-resolution image does not accurately represent the degradation caused by the camera, and a low-resolution image obtained by general image processing is used. However, there are cases where a degraded image cannot be generated by simple image processing. Therefore, a technology has been disclosed that realizes the resolution enhancement of a visible light lensless camera image by presenting the image on an LCD (liquid crystal display), photographing it with a visible light lensless camera, and generating learning data (for example, Non-Patent Document 4). In the case of a lensless camera, since the image acquired is completely different from a normal image, it is necessary to degrade the image after reconstruction in order to generate a degraded image, but this does not necessarily accurately represent the degradation caused by the camera. Therefore, as in the method of Non-Patent Document 4, it is necessary to use an actual lensless camera, and the cost of generating learning data is high. In addition, there is a need for a display that can display far-infrared images at high speed and high resolution for far-infrared lensless cameras as well.

[0005] In response to this, as a technology that can be used to increase the resolution of far-infrared images, for example, a technology has been disclosed that relates to a display in which multiple rods with diagonally cut tips are arranged two-dimensionally and the rotation angles of these rods can be individually controlled (for example, Patent Document 2). The cut surfaces at the tips of these rods are mirror-like, and by rotating around the central axis, different parts of the radiation source of a non-blackbody furnace with an arc-shaped temperature spectrum distribution can be reflected in the central axis direction. In terms of the far-infrared range, this makes it possible to change the temperature presented in the central axis direction depending on the rotation angle, and since the rotation of these rods can be controlled at high speed, it is possible to present temperature information at high speed.

[0006] Other methods that have been disclosed include using a DMD (Digital Micromirror Device) in which multiple tilted reflecting mirrors are arranged in two dimensions and the light beam is controlled by controlling the reflecting mirrors (e.g., Non-Patent Document 3), and a method in which a DMD is used to reflect infrared light from a blackbody furnace, with each reflecting mirror serving as a pixel, and the temperature being displayed for each pixel (e.g., Patent Document 1). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2001-215620 A [Patent Document 2] Patent No. 5340725 [Non-patent literature]

[0008] [Non-Patent Document 1] Dong, Chao, et al. "Image super-resolution using deep convolutional networks." IEEE transactions on pattern analysis and machine intelligence 38.2 (2015): 295-307. [Non-Patent Document 2] Jones, Lawrence E., Eric M. Olson, and George C. Goldsmith II. "Characterization measurements of the wideband infrared scene projector resistor array." Targets and Backgrounds: Characterization and Representation II. Vol. 2742. International Society for Optics and Photonics, 1996. [Non-Patent Document 3] Dudley, Dana, Walter M. Duncan, and John Slaughter. "Emerging digital micromirror device (DMD) applications." MOEMS display and imaging systems. Vol. 4985. International Society for Optics and Photonics, 2003. [Non-Patent Document 4] Khan, Salman S., et al. "Towards Photorealistic Reconstruction of Highly Multiplexed Lensless Images." Proceedings of the IEEE International Conference on Computer Vision. 2019. Summary of the Invention [Problem to be solved by the invention]

[0009] For example, according to the technology of Patent Document 1, in a specific environment, infrared light from a black body furnace is reflected by a DMD to display the temperature for each pixel.

[0010] However, the technology of Patent Document 1 does not take into account the influence of environmental temperature and disturbance. Therefore, the technology of Patent Document 1 has a problem that it is difficult to present (display) a far-infrared image with good accuracy while taking into account the influence of environmental temperature and disturbance, and the accuracy of learning about infrared images is reduced. In addition, the technology of Patent Document 2 has a problem that it is difficult to miniaturize the structure that rotates the rod due to analog control, and in the case of far-infrared rays, it is difficult to create an arc-shaped structure that changes with accurate temperature, and it is difficult to improve the temperature accuracy. Therefore, the technology of Patent Document 2 has a problem that the accuracy of learning about infrared images is reduced. Therefore, it is desired to be able to suppress the decrease in accuracy of learning about infrared images.

[0011] Therefore, the present disclosure proposes an infrared information display device, an infrared information display method, a learning data generation device, and a learning system that can suppress a decrease in accuracy of learning related to infrared images. [Means for solving the problem]

[0012] In order to solve the above problems, an infrared information display device according to one embodiment of the present disclosure includes a first blackbody furnace, a second blackbody furnace having a temperature different from that of the first blackbody furnace, a drive control unit that performs drive switching control to reflect a temperature corresponding to an inputted far-infrared image by switching a radiation light reflected by a reflector from the radiation light of the first blackbody furnace to the radiation light of the second blackbody furnace based on the temperature information of the far-infrared image, and a reflector having a plurality of reflecting mirrors arranged two-dimensionally, the reflecting mirrors corresponding to each pixel of the far-infrared image and the orientation of which is changed by the drive switching control. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a learning system according to an embodiment of the present disclosure. [Diagram 2] FIG. 11 is a diagram illustrating an example of a learning process according to an embodiment of the present disclosure. [Diagram 3] FIG. 1 is a diagram illustrating a configuration example of an information processing device according to an embodiment of the present disclosure. [Figure 4] 1 is a flowchart illustrating a procedure of a learning data generation process according to an embodiment of the present disclosure. [Diagram 5] FIG. 2 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the infrared information display device, infrared information display method, learning data generation device, and learning system according to the present application are not limited to these embodiments. In addition, in the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0015] The present disclosure will be described in the following order. 1. Embodiment 1-1. Configuration of the learning system according to the embodiment 1-2. Overview of learning process according to an embodiment of the present disclosure 1-3. Data Generation Process Procedure According to the Embodiment 2. Other embodiments 2-1.Other 3. Effects of the present disclosure 4. Hardware Configuration

[0016] [1. Embodiment] [1-1. Configuration of the learning system according to the embodiment] First, a learning system 1 shown in FIG. 1 and FIG. 2 will be described. FIG. 1 is a diagram showing a configuration example of a learning system according to an embodiment of the present disclosure. FIG. 2 is a diagram showing an example of a learning process according to an embodiment of the present disclosure. As shown in FIG. 1, the learning system 1 includes a far-infrared learning data generation device 100 and an information processing device 112. For example, the far-infrared learning data generation device 100 and the information processing device 112 may be directly connected by wire, or may be connected to be communicable by wire or wirelessly via a communication network (network N). Note that the learning system 1 shown in FIG. 1 and FIG. 2 may include a plurality of far-infrared learning data generation devices 100 and a plurality of information processing devices 112.

[0017] (Configuration of far-infrared learning data generation device) 2 is a learning data generating device that generates data used for learning about infrared rays. The far-infrared learning data generating device 100 includes a far-infrared image display device 101 and a far-infrared camera 107.

[0018] The far-infrared image display device 101 emits, toward the far-infrared camera 107, radiant light corresponding to a presentation image 111, which is a far-infrared image of teacher data input from an information processing device 112. The far-infrared camera 107 observes the radiant light emitted from the far-infrared image display device 101 to generate an observation image 110.

[0019] (Explanation of the configuration of a far-infrared image display device) The far-infrared image display device 101 includes a blackbody furnace 102, a blackbody furnace 103, a DMD 104, a DMD driver 105, and a lens .

[0020] Here, the blackbody furnace 102 and the blackbody furnace 103 are temperature devices that have an extremely uniform temperature distribution in a plane and can reproduce a set temperature with high accuracy. The temperature set in the blackbody furnace 102 is different from the temperature set in the blackbody furnace 103. For example, the temperature of the blackbody furnace 103 may be set lower than the temperature of the blackbody furnace 102, or vice versa. The DMD 104, the blackbody furnace 102, and the blackbody furnace 103 are arranged at positions where radiation from either the blackbody furnace 102 or the blackbody furnace 103 is observed when observed from a specific observation direction. The specific observation direction here is, for example, a direction based on the positional relationship between the far-infrared camera 107 that observes radiation from the DMD 104 and the DMD 104.

[0021] The DMD driver 105 obtains a presentation image 111 of a far-infrared image, which is teacher data, from an information processing device 112. The DMD driver 105 calculates temperature information corresponding to each pixel from luminance information of each pixel of the presentation image 111, and transmits a drive switching control signal for controlling the reflecting mirror of the DMD 104 based on the temperature information. The control is performed, for example, by PWM (Pulse Width Modulation) control. The DMD driver 105 expresses a temperature between the blackbody furnace 102 and the blackbody furnace 103 by performing PWM control on the DMD 104 in consideration of weighted addition based on a duty ratio.

[0022] The DMD 104 receives a drive switching control signal from the DMD driver 105. The DMD 104 reproduces the temperature corresponding to each pixel of the presentation image 111 by switching the orientation of each reflecting mirror based on the drive switching control, and radiates the temperature to the far-infrared camera 107.

[0023] Specifically, the DMD 104 switches the orientation of each reflector to either an orientation in which the radiation from the black body furnace 102 is observed from a specific observation direction or an orientation in which the radiation from the black body furnace 103 is observed from a specific observation direction based on the drive switching control signal. At this time, each reflector serving as each pixel of the DMD 104 can be switched at high speed, similar to that disclosed in Non-Patent Document 3, so that it is possible to perform gradation display for a presented video in which a presented image is arranged in time series as one frame. When the DMD 104 is a DMD capable of switching, for example, 10,000 times / second and displays an image at 30 fps per frame, it is possible to switch the orientation of the reflector 333 times within the time of one frame, making it possible to express 333 gradations of light and shade. For example, the presented image 111 is one frame in the video.

[0024] If the frame rate of the image displayed by the DMD 104 is f, one period T for displaying the image is T=1 / f. The following describes an example in which the temperature of the blackbody furnace 102 is t1, the temperature of the blackbody furnace 103 is t2, and the temperature to be displayed by a pixel x in the presented image 111 is t(x). In this case, the pixel corresponding to pixel x in the observed image 110 observed by the far-infrared camera 107 is x'. The relationship between x and x' can be expressed by the following formula (1).

[0025]

number

[0026] During one period T, the ratio of time that the pixel of the DMD 104 corresponding to pixel x reflects light from the blackbody furnace 102 is α(x), and the ratio of time that the pixel reflects light from the blackbody furnace 103 is (1-α(x)). Then, the temperature t C (x´) is expressed as follows in equation (2).

[0027]

number

[0028] Temperature t shown in equation (2) C (light corresponding to) is collected by lens 106, and the light from the mirror that hits a pixel on DMD 104 is collected on a pixel of FPA 109 of far-infrared camera 107. Note that α(x) satisfies the following formula (3).

[0029]

number

[0030] t(x)=t C By determining α(x) so that (x´) is obtained, DMD 104 is able to display the desired temperature for each pixel. Here, the relationship between the switching of the mirror of DMD 104 and α(x) will be explained. The maximum number of times that the mirror of DMD 104 can be switched between a first direction and a second direction different from the first direction during time T (one period T) is defined as n. In this case, the time t1 during which the mirror of DMD 104 faces the first direction can be expressed by the following equation (4), and the time t2 during which the mirror of DMD 104 faces the second direction can be expressed by the following equation (5).

[0031]

number

[0032]

number

[0033] In order to minimize flicker when observed by the far-infrared camera 107, the DMD 104 should perform switching as many times as possible within the range where formulas (4) and (5) hold, and perform switching so that the distribution of switching in the time direction is as uniform as possible. Therefore, the DMD driver 105 executes drive switching control of the DMD 104 so that the number of switching times is large (for example, the largest) within the range where formulas (4) and (5) hold, and the distribution of switching in the time direction is uniform.

[0034] As a result, the temperature corresponding to each pixel of the presented image 111 is reproduced using the temperature range between the temperature of the blackbody furnace 102 and the temperature of the blackbody furnace 103. Note that the details of the configuration of the DMD 104 are similar to those disclosed in Non-Patent Document 3, and therefore will not be described.

[0035] The lens 106 is an optical element disposed between the DMD 104 and the far-infrared camera 107. The shape of the lens 106 is not limited to the shape shown in Fig. 2, and may be any shape.

[0036] (Explanation of the configuration of a far-infrared camera) The far-infrared camera 107 includes a lens 108 and an FPA 109 (Focal Plane Array). The far-infrared camera 107 observes the radiation light from the far-infrared image display device 101, and generates an observation image 110 corresponding to a presented image 111. The lens 108 is an optical element disposed between the far-infrared image display device 101 and the FPA 109. In the example of FIG. 2, the lens 108 is disposed with its optical axis aligned with that of the lens 106. The shape of the lens 108 is not limited to the shape shown in FIG. 2, and may be any shape. The FPA 109 is a so-called multi-element sensor, and is a two-dimensional sensor. The far-infrared camera 107 is not limited to the FPA 109, and may have any sensor (image sensor, etc.) as long as it can generate a desired observation image.

[0037] In the example of FIG. 2, the DMD 104, the lens 106, the lens 108, and the FPA 109 are arranged with their centers aligned.

[0038] (Description of the configuration of information processing device) The information processing device 112 is an information processing device that executes information processing according to the embodiment. The information processing device 112 is a terminal device used by a user such as an administrator of the learning system 1. For example, the information processing device 112 may be various devices used by users, such as a smartphone, a tablet terminal, a smart speaker, a notebook PC (Personal Computer), a desktop PC, a mobile phone, or a PDA (Personal Digital Assistant). The configuration of the information processing device 112 will be described below with reference to FIG. 3. FIG. 3 is a diagram showing an example of the configuration of an information processing device according to the embodiment of the present disclosure.

[0039] 3, the information processing device 112 includes a communication unit 11, an input unit 12, a display unit 13, a storage unit 14, and a control unit 15. In the example of Fig. 3, the information processing device 112 includes the input unit 12 (e.g., a keyboard, a mouse, etc.) that receives various operations from an administrator of the information processing device 112, etc., and the display unit 13 (e.g., a liquid crystal display, etc.) that displays various information.

[0040] The communication unit 11 is realized by, for example, a NIC (Network Interface Card), a communication circuit, etc. The communication unit 11 is connected to a network N (such as the Internet) by wire or wirelessly, and transmits and receives information to and from other devices such as the far-infrared learning data generation device 100 via the network N.

[0041] The input unit 12 receives input such as a user's selection of a learning method, etc. The input unit 12 may receive input via a keyboard, a mouse, or a touch panel provided in the information processing device 112.

[0042] The display unit 13 is a display device (display unit) such as a display, and displays various information such as the information generated by the generation unit 154. The display unit 13 may be the display 114 in FIG. 2. In this case, the display unit 13 may be a separate entity from the information processing device 112.

[0043] Furthermore, the information processing device 112 may have a functional configuration for outputting information, in addition to the display unit 13. The information processing device 112 may have an audio output unit such as a speaker that outputs information as audio.

[0044] The storage unit 14 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. The storage unit 14 has a teacher data storage unit 141 and a model information storage unit 142. Note that the storage unit 14 stores various information, including, but not limited to, the above, such as a presentation image list including a plurality of presentation images.

[0045] The teacher data storage unit 141 stores various information related to data used for learning. The teacher data storage unit 141 stores a teacher data list used for learning. The teacher data storage unit 141 stores a combination (set) of an observed image 110 as input data and a presentation image 111 corresponding to the observed image 110 as answer information as teacher data.

[0046] The teacher data storage unit 141 is not limited to the above, and may store various kinds of information depending on the purpose.

[0047] The model information storage unit 142 stores information about a model. For example, the model information storage unit 142 stores information (model data) indicating the structure of a model (network). For example, the model information storage unit 142 includes items such as "model ID", "purpose", and "model data".

[0048] "Model ID" indicates identification information for identifying a model. "Use" indicates the use of the corresponding model. "Model Data" indicates the data of the model. "Model Data" includes various information that constitutes the model, such as information about the network included in the model and functions.

[0049] The model information storage unit 142 stores the neural network 113 in association with the model ID "M1". The model information storage unit 142 also stores the purpose and model data of the neural network 113. The model information storage unit 142 may store multiple models. The model information storage unit 142 may store various models (networks) other than the neural network 113.

[0050] The model information storage unit 142 is not limited to the above, and may store various types of information depending on the purpose.

[0051] The control unit 15 is realized, for example, by a central processing unit (CPU), a micro processing unit (MPU), or the like executing a program (for example, an information processing program according to the present disclosure) stored in the information processing device 112 using a RAM or the like as a working area. The control unit 15 may also be realized, for example, by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0052] 3, the control unit 15 has an acquisition unit 151, a learning unit 152, an extraction unit 153, a generation unit 154, and a transmission unit 155, and realizes or executes the functions and actions of information processing described below. Note that the internal configuration of the control unit 15 is not limited to the configuration shown in FIG. 3, and may be other configurations as long as they perform the information processing described below.

[0053] The acquisition unit 151 acquires various pieces of information from the storage unit 14. The acquisition unit 151 acquires information accepted by the input unit 12.

[0054] The acquiring unit 151 acquires an observation image observed by the far-infrared learning data generating device 100. The acquiring unit 151 stores the acquired observation image in the teacher data storage unit 141 as learning data (teacher data).

[0055] The learning unit 152 learns a model so that, when a low-resolution image is input, a high-resolution image corresponding to the input image is output. The learning unit 152 learns parameters of the neural network 113 and generates the neural network 113.

[0056] The learning unit 152 learns the neural network 113 by using the teacher data stored in the teacher data storage unit 141. The learning unit 152 stores the trained neural network 113 in the model information storage unit 142.

[0057] For example, when an observed image 110 that is a degraded image influenced by the characteristics of the lens 106, the lens 108, the far-infrared camera 107, etc. is input, the learning unit 152 learns parameters of the neural network 113 so as to output a presented image 111 corresponding to the observed image 110. The learning unit 152 generates the neural network 113, which is, for example, a DNN (Deep Neural Network) such as a CNN (Convolutional Neural Network).

[0058] The extraction unit 153 extracts a presentation image from the presentation image list stored in the storage unit 14 .

[0059] The generation unit 154 generates teacher data that associates an observed image with a presentation image that corresponds to the observed image. The generation unit 154 stores the generated teacher data in the teacher data storage unit 141.

[0060] The transmitting unit 155 transmits various information to an external information processing device. The transmitting unit 155 transmits information stored in the storage unit 14. The transmitting unit 155 transmits the presentation image extracted by the extracting unit 153 to the far-infrared learning data generation device 100.

[0061] [1-2. Overview of learning process according to an embodiment of the present disclosure] From here, the learning process shown in Fig. 2 will be described. The learning process according to the embodiment of the present disclosure is realized by a learning system 1. The presented image 111 in Fig. 2 is a so-called monochrome image. The presented image 111 is an image in which temperature is converted into a brightness value, and for example, the higher the temperature, the closer to white the presented image 111 is, and the lower the temperature, the closer to black it is.

[0062] First, the information processing device 112 determines the presentation image 111 to be input to the far-infrared learning data generation device 100, and extracts the presentation image 111 from the presentation image list (step S1).

[0063] The information processing device 112 inputs the presentation image 111 to the far-infrared learning data generation device 100 (step S2). Specifically, the information processing device 112 inputs the presentation image 111 to the DMD driver 105.

[0064] The DMD driver 105 expresses a temperature corresponding to the input presentation image 111 by controlling the DMD 104 for each pixel of the presentation image 111 (step S3).

[0065] The far-infrared learning data generating device 100 observes the image displayed by the far-infrared image display device 101 with the far-infrared camera 107, obtains an observed image 110, and outputs the observed image 110 to the information processing device 112 (step S4).

[0066] Then, the information processing device 112 performs a learning process by providing the observed image 110 and the presented image 111 to the neural network 113 as learning data (step S5). The neural network 113 is a model (network) that receives an image as an input and outputs (generates) an image corresponding to the input image. For example, the information processing device 112 inputs the observed image 110 to the neural network 113 as learning data, and the neural network 113 uses the presented image 111 as teacher data to perform the learning process. For example, the information processing device 112 performs the learning process using a method disclosed in Non-Patent Document 1. As a result, the information processing device 112 can generate a neural network 113 that receives an infrared image as an input and outputs an infrared image with improved image quality.

[0067] As described above, the information processing device 112 provides the neural network 113 with the observed image 110 as learning data and the presented image 111 as teacher data, thereby enabling processing such as high-resolution image restoration of the far-infrared camera to be realized. In addition, the light observed by the far-infrared camera 107 is restricted to either the blackbody furnace 102 or the blackbody furnace 103, and is less susceptible to the effects of temperatures of other places, such as the housing (of the far-infrared image display device 101), and is therefore less susceptible to disturbances. In other words, the far-infrared image display device 101 can reduce the room for the effects of environmental temperature and disturbances compared to a case with one blackbody furnace.

[0068] As a result, the far-infrared learning data generation device 100 can generate learning data that is less affected by environmental temperature and disturbances compared to the case where one blackbody furnace is used, and can suppress deterioration in learning accuracy regarding infrared images.

[0069] [1-3. Procedure of data generation process according to embodiment] Next, a procedure of a data generation process according to an embodiment will be described with reference to Fig. 4. Fig. 4 is a flowchart showing a procedure of a learning data generation process according to an embodiment of the present disclosure.

[0070] 4, the far-infrared image display device 101 of the far-infrared learning data generating device 100 sets two blackbody furnaces to different temperatures t1 and t2 (step S101). For example, the far-infrared image display device 101 sets the blackbody furnace 102 to temperature t1 and the blackbody furnace 103 to temperature t2. Since the presentable temperatures are between temperature t1, which is the temperature of the blackbody furnace 102, and temperature t2, which is the temperature of the blackbody furnace 103, the far-infrared image display device 101 determines the temperatures t1 and t2 based on the range of temperatures to be displayed.

[0071] Then, the information processing device 112 empties the teacher data list for result output (step S102).

[0072] Then, the information processing device 112 extracts one presentation image from the presentation image list (step S103).

[0073] If the information processing device 112 has not finished extracting all the presentation images from the presentation image list (step S104: No), it outputs the presentation image to the DMD driver 105 (step S105). For example, the information processing device 112 transmits to the DMD driver 105 a presentation image that represents temperature by the shade of each pixel.

[0074] The DMD driver 105 of the far-infrared image display device 101 performs PWM control of the DMD 104 in accordance with the density of the input presentation image (step S106).

[0075] The far-infrared camera 107 observes the light from the DMD 104 and outputs an observed image (step S107).

[0076] The information processing device 112 adds the obtained observed image and the presented image to a teacher data list (step S108).

[0077] If all presentation images have already been extracted from the presentation image list (step S104: Yes), the information processing device 112 outputs the teacher data and ends the process (step S109).

[0078] The information processing device 112 can construct a network that outputs an ideal image from a camera image by learning a network such as a CNN (such as the neural network 113) using the generated teacher data list.

[0079] [2. Other embodiments] The processing according to each of the above-mentioned embodiments may be implemented in various different forms (modified forms) other than the above-mentioned embodiments. For example, the system configuration is not limited to the above-mentioned examples, and may be in various forms. For example, the information processing device 112 and the far-infrared learning data generation device 100 may be integrated. Note that the above is an example, and the learning system may be realized in various configurations.

[0080] [2-1.Other] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0081] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0082] Furthermore, the above-described embodiments and modifications can be appropriately combined as long as the processing contents are not contradictory.

[0083] Furthermore, the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained.

[0084] [3. Effects of this disclosure] As described above, the infrared information display device (far-infrared image display device 101 in the embodiment) according to the present disclosure includes a first blackbody furnace (blackbody furnace 102 in the embodiment), a second blackbody furnace (blackbody furnace 103 in the embodiment) having a temperature different from that of the first blackbody furnace, a drive control unit (DMD driver 105 in the embodiment), and a reflector (DMD 104 in the embodiment). The drive control unit performs drive switching control to reflect a temperature corresponding to the temperature information by switching the radiation reflected by the reflector from the radiation of the first blackbody furnace to the radiation of the second blackbody furnace based on the temperature information of the input far-infrared image. The reflector has a plurality of reflectors arranged two-dimensionally, which correspond to each pixel of the far-infrared image and whose orientation is changed by drive switching control.

[0085] As a result, the infrared information display device according to the present disclosure can reflect the radiation light from one of the two blackbody furnaces with a specified temperature by switching the radiation light reflected by the multiple reflecting mirrors in the two blackbody furnaces with different temperatures, and can narrow the room for the influence of environmental temperature and disturbance compared to the case of using one blackbody furnace. In addition, since the temperature of each blackbody furnace is controlled with high precision, the infrared information display device can display a light synthesized from only the reflected light from the blackbody furnace, thereby displaying a highly accurate far-infrared image. As a result, the infrared information display device can generate appropriate learning data by an observation device such as a far-infrared camera that observes the radiation light from the infrared information display device. When learning is performed using the learning data, highly accurate learning is possible. Therefore, the infrared information display device can suppress a decrease in the accuracy of learning regarding the infrared image.

[0086] Moreover, the positional relationship of the reflector, the first blackbody furnace, and the second blackbody furnace is such that, when observed from a specific observation direction based on the positional relationship between the reflector and a far-infrared camera that observes the radiated light from the reflector, radiated light from either the first blackbody furnace or the second blackbody furnace is observed in response to drive switching control. As a result, the infrared information display device is arranged so that radiated light from either of the two blackbody furnaces is observed from a specific observation direction based on the positional relationship between the far-infrared camera and the reflector, thereby narrowing the room for influence of environmental temperature and disturbance compared to the case where there is only one blackbody furnace. Therefore, the infrared information display device can suppress a decrease in accuracy of learning regarding infrared images.

[0087] Moreover, the reflecting unit displays a temperature between the first temperature of the first blackbody furnace and the second temperature of the second blackbody furnace by drive switching control. As a result, the infrared information display device can display a temperature between the first temperature and the second temperature by using two blackbody furnaces, the first blackbody furnace and the second blackbody furnace, so that it is possible to reduce the room for influence of environmental temperature and disturbance compared to the case where there is one blackbody furnace. Therefore, the infrared information display device can suppress a decrease in accuracy of learning regarding infrared images.

[0088] The drive control unit also performs drive switching control to switch the orientation of the multiple reflectors to either a first direction in which the radiation from the first blackbody furnace is observed from a specific observation direction, or a second direction in which the radiation from the second blackbody furnace is observed from a specific observation direction. This allows the infrared information display device to control the orientation of the reflectors so that the radiation from either the first blackbody furnace or the second blackbody furnace is observed, thereby narrowing the room for the influence of environmental temperature and disturbance compared to the case where there is only one blackbody furnace. Therefore, the infrared information display device can suppress a decrease in accuracy of learning regarding infrared images.

[0089] Moreover, the learning data generation device according to the present disclosure (far-infrared learning data generation device 100 in the embodiment) includes an infrared information display device and a far-infrared camera. The infrared information display device reflects a temperature corresponding to the temperature information by switching the radiant light reflected by a first blackbody furnace and a second blackbody furnace having a temperature different from that of the first blackbody furnace, which are arranged two-dimensionally and correspond to each pixel of the far-infrared image, based on the temperature information of the input far-infrared image. The far-infrared camera observes the radiant light from the infrared information display device and generates an observation image.

[0090] As a result, the learning data generation device according to the present disclosure can generate learning data that is less affected by environmental temperature and disturbances than when a single blackbody furnace is used, by observing the radiation reflected by multiple reflectors in two blackbody furnaces with different temperatures and generating observed images. When learning is performed using the learning data, learning with high accuracy is possible. Therefore, the learning data generation device can suppress a decrease in accuracy of learning regarding infrared images.

[0091] Moreover, a learning system according to the present disclosure (learning system 1 in the embodiment) includes an infrared information display device, a far-infrared camera, and an information processing device. The infrared information display device reflects a temperature corresponding to the temperature information by switching the radiant light reflected by a first blackbody furnace and a second blackbody furnace having a temperature different from the first blackbody furnace, which are arranged two-dimensionally and correspond to each pixel of the far-infrared image, based on the temperature information of the input far-infrared image. The far-infrared camera observes the radiant light from the infrared information display device and generates an observation image. The information processing device performs machine learning using a combination of the far-infrared image and the observation image.

[0092] As a result, the learning system according to the present disclosure performs machine learning using observation images generated by observing synchrotron radiation reflected by multiple reflectors in two blackbody furnaces with different temperatures and the far-infrared images on which the observation images are based, thereby making it possible to perform machine learning using data that is less affected by environmental temperature and disturbances compared to the case of using one blackbody furnace. This allows the learning system to perform learning with high accuracy. Therefore, the learning system can suppress a decrease in accuracy of learning regarding infrared images.

[0093] [4. Hardware configuration] Information devices such as the information processing device 112 according to each embodiment described above are realized by a computer 1000 having a configuration as shown in Fig. 5, for example. Fig. 5 is a hardware configuration diagram showing an example of a computer that realizes the functions of the information devices. The information processing device 112 according to the embodiment will be described below as an example. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. Each unit of the computer 1000 is connected by a bus 1050.

[0094] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0095] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0096] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records an information processing program according to the present disclosure, which is an example of program data 1450.

[0097] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0098] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0099] For example, when the computer 1000 functions as the information processing device 112 according to the embodiment, the CPU 1100 of the computer 1000 executes an information processing program loaded onto the RAM 1200 to realize functions of the control unit 15, etc. Also, the information processing program according to the present disclosure and data in the storage unit 14 are stored in the HDD 1400. Note that the CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, these programs may be obtained from other devices via an external network 1550.

[0100] The present technology can also be configured as follows. (1) a first blackbody furnace; a second blackbody furnace having a temperature different from that of the first blackbody furnace; a drive control unit that performs drive switching control to switch the radiation light reflected by the reflecting mirror from the radiation light of the first blackbody furnace to the radiation light of the second blackbody furnace based on temperature information of the input far-infrared image, thereby reflecting a temperature corresponding to the temperature information; a reflecting unit having a plurality of reflecting mirrors arranged two-dimensionally, the reflecting mirrors corresponding to each pixel of the far-infrared image and the orientation of which is changed by the drive switching control; An infrared information display device comprising: (2) an arrangement relationship between the reflecting unit, the first blackbody furnace, and the second blackbody furnace is such that, when observed from a specific observation direction based on a positional relationship between the reflecting unit and a far-infrared camera that observes the radiated light from the reflecting unit, radiated light from either the first blackbody furnace or the second blackbody furnace is observed in response to the drive switching control. An infrared information display device according to (1). (3) The reflecting portion is and displaying a temperature between a first temperature of the first blackbody furnace and a second temperature of the second blackbody furnace by the driving switching control. An infrared information display device according to (2). (4) The drive control unit is performing the drive switching control to switch the orientation of the plurality of reflecting mirrors to either a first direction in which the radiation from the first blackbody furnace is observed from the specific observation direction or a second direction in which the radiation from the second blackbody furnace is observed from the specific observation direction; An infrared information display device according to (2) or (3). (5) based on temperature information of the input far-infrared image, switching the radiation reflected by the reflecting mirror from the radiation of a first blackbody furnace to the radiation of a second blackbody furnace having a temperature different from that of the first blackbody furnace, thereby performing drive switching control to reflect a temperature corresponding to the temperature information; changing orientations of a plurality of reflecting mirrors arranged two-dimensionally and corresponding to each pixel of the far-infrared image by the drive switching control; A method for displaying infrared information that performs processing. (6) an infrared information display device that reflects a temperature corresponding to the input temperature information of a far-infrared image by switching between a first blackbody furnace and a second blackbody furnace having a temperature different from that of the first blackbody furnace, which are two-dimensionally arranged and reflect radiation light by a plurality of reflecting mirrors corresponding to each pixel of the far-infrared image, based on the temperature information of the input far-infrared image; a far-infrared camera that observes the radiation from the infrared information display device and generates an observation image; A learning data generating device comprising: (11) an infrared information display device that reflects a temperature corresponding to the input temperature information of a far-infrared image by switching between a first blackbody furnace and a second blackbody furnace having a temperature different from that of the first blackbody furnace, which are two-dimensionally arranged and reflect radiation light by a plurality of reflecting mirrors corresponding to each pixel of the far-infrared image, based on the temperature information of the input far-infrared image; a far-infrared camera that observes the radiation from the infrared information display device and generates an observation image; an information processing device that performs machine learning using a combination of the far-infrared image and the observation image; A learning system comprising: [Explanation of symbols]

[0101] 1. Learning System 100 Far-infrared learning data generating device (learning data generating device) 101 Far-infrared image display device (infrared information display device) 102 Blackbody Furnace (First Blackbody Furnace) 103 Blackbody Furnace (Second Blackbody Furnace) 104 DMD (Reflector) 105 DMD driver (drive control unit) 106 Lens 107 Far-infrared camera 108 Lens 109 FPA (image sensor) 112 Information processing equipment 11 Communications Department 12 Input section 13 Display unit (display) 14 Storage section 141 Teacher data storage unit 142 Model information storage unit 15 Control section 151 Acquisition Department 152 Learning Department 153 Extraction part 154 Generation part 155 Transmitter

Claims

1. a first blackbody furnace; a second blackbody furnace having a temperature different from that of the first blackbody furnace; a drive control unit that performs drive switching control to switch the radiation light reflected by the reflecting mirror from the radiation light of the first blackbody furnace to the radiation light of the second blackbody furnace based on temperature information of the input far-infrared image, thereby reflecting a temperature corresponding to the temperature information; a reflecting unit having a plurality of reflecting mirrors arranged two-dimensionally, the reflecting mirrors corresponding to each pixel of the far-infrared image and the orientation of which is changed by the drive switching control; An infrared information display device comprising:

2. an arrangement relationship between the reflecting unit, the first blackbody furnace, and the second blackbody furnace is such that, when observed from a specific observation direction based on a positional relationship between the reflecting unit and a far-infrared camera that observes the radiated light from the reflecting unit, radiated light from either the first blackbody furnace or the second blackbody furnace is observed in response to the drive switching control.

2. An infrared information display device according to claim 1.

3. The reflecting portion is and displaying a temperature between a first temperature of the first blackbody furnace and a second temperature of the second blackbody furnace by the driving switching control.

3. An infrared information display device according to claim 2.

4. The drive control unit is performing the drive switching control to switch the orientations of the plurality of reflecting mirrors to either a first direction in which the radiation from the first blackbody furnace is observed from the specific observation direction or a second direction in which the radiation from the second blackbody furnace is observed from the specific observation direction; 3. An infrared information display device according to claim 2.

5. based on temperature information of the input far-infrared image, switching the radiation reflected by the reflecting mirror from the radiation of a first blackbody furnace to the radiation of a second blackbody furnace having a temperature different from that of the first blackbody furnace, thereby performing drive switching control to reflect a temperature corresponding to the temperature information; changing orientations of a plurality of reflecting mirrors arranged two-dimensionally and corresponding to each pixel of the far-infrared image by the drive switching control; A method for displaying infrared information that performs processing.

6. an infrared information display device that reflects a temperature corresponding to the input temperature information of a far-infrared image by switching between a first blackbody furnace and a second blackbody furnace having a temperature different from that of the first blackbody furnace, which are two-dimensionally arranged and reflect radiation light by a plurality of reflecting mirrors corresponding to each pixel of the far-infrared image, based on the temperature information of the input far-infrared image; a far-infrared camera that observes the radiation from the infrared information display device and generates an observation image; A learning data generating device comprising:

7. an infrared information display device that reflects a temperature corresponding to the input temperature information of a far-infrared image by switching between a first blackbody furnace and a second blackbody furnace having a temperature different from that of the first blackbody furnace, which are two-dimensionally arranged and reflect radiation light by a plurality of reflecting mirrors corresponding to each pixel of the far-infrared image, based on the temperature information of the input far-infrared image; a far-infrared camera that observes the radiation from the infrared information display device and generates an observation image; an information processing device that performs machine learning using a combination of the far-infrared image and the observation image; A learning system comprising:

Images