Imaging apparatus, method for controlling imaging apparatus, and program

Abstract

To provide an imaging apparatus, a method for controlling the imaging apparatus, and a program that can record temperature information of a subject that is appropriately narrowed down.SOLUTION: An imaging apparatus 100 comprises: a visible light image pickup device 103 that acquires a visible light image; a non-visible light image pickup device 104 that acquires a non-visible light image; an object detection unit 116 that detects one or more subjects on the basis of the visible light image to acquire one or more detection results; an object identification unit 117 that identifies the types of the one or more subjects detected by the object detection unit 116 on the basis of the visible light image to acquire one or more identification results; a temperature recording determination unit 120 that determines whether to record temperature information of the subject on the basis of the detection results and the identification results; and a temperature acquisition unit 121 and a recording unit 122 that, when the temperature recording determination unit 120 determines to record the temperature information of the subject, acquire the temperature information of the subject determined to be recorded on the basis of the non-visible image and the detection results, and record the temperature information in a recording medium 107.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an imaging device, a control method for an imaging device, and a program. [Background technology]

[0002] In recent years, imaging devices capable of simultaneously acquiring visible light and invisible light have been developed. For example, Patent Document 1 discloses a technique for acquiring subject information not shown in a visible light image based on the type of subject, and performing effect processing on the image. In addition, in recent years, a hot and cold sensation presentation device has appeared that allows a user to enjoy a photograph not only visually but also tactilely or thermally. As such a hot and cold sensation presentation device, for example, Non-Patent Document 1 discloses a display in which Peltier elements capable of switching between cooling and heating are arranged in a matrix-like area, and an image can be displayed. The display described in Non-Patent Document 1 can express temperature by controlling each Peltier element while displaying an image, thereby cooling or heating the display. As a result, a user using the display can touch the display and experience coldness or warmth, that is, a hot and cold sensation is presented. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6819670 [Non-patent literature]

[0004] [Non-Patent Document 1] Kumiko Kushiyama, Yuki Doi, Shinji Sasada, and Noriaki Baba, "Thermo Drawing: Development of a temperature drawing system using a small tactile display with cold and hot display," Information Processing Society of Japan Interaction 2012, March 16, 2012, pp.723-728 Summary of the Invention [Problem to be solved by the invention]

[0005] Incidentally, the presentation of hot and cold sensation in a hot and cold sensation presentation device may be performed based on subject information acquired and recorded by an imaging device using visible light and temperature information acquired and recorded using invisible light. In such a case, when the hot and cold sensation presentation device tries to limit the subject to which the hot and cold sensation is presented, the temperature information acquired and recorded without controlling the imaging device becomes redundant information for the hot and cold sensation presentation device. Therefore, there is a problem in that the hot and cold sensation presentation device is required to perform high-spec post-processing such as detection required to limit the subject.

[0006] The present invention has been made in view of the above problems, and an object of the present invention is to provide an imaging device, a control method for an imaging device, and a program that can record temperature information about a subject that has been appropriately focused. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the imaging device of the present invention is characterized in comprising a first imaging means for acquiring a visible light image, a second imaging means for acquiring an invisible light image, a detection means for acquiring one or more detection results by detecting one or more subjects based on the visible light image, an identification means for acquiring one or more identification results by identifying the types of the one or more subjects detected by the detection means based on the visible light image, a recording determination means for determining whether or not to record temperature information of the subject based on the detection results and the identification results, and an acquisition and recording means for, if the recording determination means determines that the temperature information of the subject should be recorded, acquiring temperature information of the subject determined by the recording determination means based on the invisible light image and the detection results and recording it on a recording medium. Effect of the Invention

[0008] According to the present invention, it is possible to record temperature information about a subject with an appropriately focused aperture. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an imaging device according to a first embodiment. [Figure 2A] FIG. 13 is a diagram showing the display unit displaying a display image on which a temperature recording frame is superimposed. [Figure 2B] FIG. 13 is a diagram showing the display unit displaying a display image on which a temperature recording icon is superimposed. [Diagram 3] 5 is a flowchart showing a temperature recording control in the first embodiment. [Figure 4] FIG. 2 is a diagram showing a table in which the detection results of the object detection unit, the identification results of the object identification unit, and the judgment results of the temperature recording and judgment unit correspond to each other for five scenes assumed in the first embodiment. [Diagram 5] 10 is a flowchart showing a temperature recording control in the second embodiment. [Figure 6] FIG. 13 is a diagram showing a display unit displaying a display image with a selection frame superimposed thereon. [Figure 7] 13 is a flowchart showing a temperature recording control in the third embodiment. [Figure 8] FIG. 13 is a diagram showing a list of priority type settings assumed in the third embodiment. [Figure 9] 13 is a flowchart showing a temperature recording control in the fourth embodiment. [Figure 10] FIG. 2 is a conceptual diagram showing a state in which the first thermal sensation providing device is used. [Figure 11] FIG. 13 is a conceptual diagram showing a state in which the second thermal sensation presentation device is used. [Figure 12] FIG. 13 is a conceptual diagram showing a state in which the third thermal sensation presentation device is used. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Each embodiment of the present invention will be described in detail below with reference to the drawings. However, the configurations described in each of the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exert a similar function. In addition, any configuration may be added. In addition, any two or more configurations (features) of each embodiment can be combined.

[0011] The imaging device according to each embodiment of the present invention acquires temperature information of a subject, and records the acquired temperature information of the subject in association with image information indicating the center position and area in the image of the subject. The temperature information and image information of the subject acquired and recorded in this manner can be used as control data for a hot and cold sensation presentation device to present the hot and cold sensation of the subject on its display image, etc. The hot and cold sensation presentation device may receive the temperature information and image information of the subject acquired and recorded by the imaging device via a recording medium, or via communication with the imaging device. In the following, first, the imaging device will be described in each embodiment, and then the hot and cold sensation presentation device will be described.

[0012] First Embodiment Hereinafter, the first embodiment will be described with reference to Figs. 1 to 4. Fig. 1 is a block diagram showing the configuration of an imaging device 100 according to the first embodiment. The imaging device 100 has an optical system 101, a shutter 102, a visible light imaging element 103 and a non-visible light imaging element 104 as an imaging unit, an image processing unit 105, a display unit 106, a recording medium 107, and a system control unit 108. The imaging device 100 further has a release button 109, a mode changeover switch 110, an operation unit 111, a non-volatile memory 112, and a system memory 113. In addition, the image processing unit 105 has an image pre-processing unit 114, a memory 115, an object detection unit 116, an object identification unit 117, a tracking unit 118, an image post-processing unit 119, a temperature recording and determination unit 120, a temperature acquisition unit 121, a recording unit 122, and a display frame superimposing unit 123.

[0013] The optical system 101 forms an image of visible light and non-visible light from a subject on the visible light imaging element 103 and the non-visible light imaging element 104. The optical system 101 is composed of, for example, a plurality of lenses, an aperture, etc., and a driving device (not shown) performs zoom control, focus control, aperture control, etc. The shutter 102 adjusts the exposure time of the visible light imaging element 103 and the non-visible light imaging element 104 by controlling the shutter speed by the system control unit 108.

[0014] The visible light imaging element 103 (first imaging means) converts visible light focused via the optical system 101 into an electrical signal to obtain a digital image, and outputs the digital image to an image pre-processing unit 114 of the image processing unit 105. The visible light imaging element 103 is realized, for example, by using a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The CMOS image sensor has, for example, a color filter in an RGB Bayer array. In the following description, a digital image obtained from visible light is referred to as a "visible light image."

[0015] The non-visible light imaging element 104 (second imaging means) converts the non-visible light incident via the optical system 101 into an electrical signal to obtain a digital image, and outputs the digital image to the image pre-processing unit 114 of the image processing unit 105. The non-visible light imaging element 104 is realized, for example, by using a CMOS image sensor having a color filter that transmits the wavelength range of near-infrared light. In the following description, a digital image obtained from non-visible light is referred to as a "non-visible light image." In the first embodiment, the optical system 101 and the shutter 102 are configured to be shared by the visible light imaging element 103 and the non-visible light imaging element 104, but a configuration in which the optical system 101 and the shutter 102 are provided separately for the visible light imaging element 103 and the non-visible light imaging element 104 may also be used.

[0016] The image processing unit 105 performs, for example, image processing for display and image processing for detection and tracking using the visible light image. Among the components of the image processing unit 105, the image pre-processing unit 114 converts the visible light image into an RGB signal or a YUV signal and writes it to the memory 115. The image pre-processing unit 114 also writes the non-visible light image to the memory 115. For the memory 115, a dynamic random access memory (DRAM) or a static random access memory (SRAM) is used.

[0017] The object detection unit 116 detects one or more objects from the RGB signal or YUV signal for each frame. Furthermore, the object detection unit 116 (detection means) acquires, as detection results, number information indicating how many objects have been detected, size information indicating the size, center position information indicating the center position, area information indicating the area, and the like, for the one or more objects detected. The object identification unit 117 (identification means) acquires, as an identification result, class information indicating the type of the identified object by identifying the object detected by the object detection unit 116. The object identification unit 117 also has an authentication function, which will be described later.

[0018] The object detection unit 116 and the object identification unit 117 are realized by a classifier using machine learning such as a random forest or a neural network. The classifier can detect one or more objects by learning in advance using learning data. In addition, the classifier can acquire the above-mentioned number information and class information, such as how many people are included and how many cats are included, for one or more detected objects by learning in advance using learning data. Furthermore, the classifier can acquire the above-mentioned size information, center position information, and area information, as well as a reliability score, for one or more detected objects. In this way, the object detection unit 116 and the object identification unit 117 can quickly perform processing based on the visible light image by using machine learning, even if the visible light image is a complex scene. Note that the object detection unit 116 and the object identification unit 117 may be configured by a single element (detection and identification means) that combines the functions of each. This modified example can reduce the number of components of the image processing unit 105 compared to the first embodiment. In contrast, in the first embodiment, the object detection unit 116 and the object identification unit 117 are separate components, and therefore the processing load is reduced and the processing speed is improved compared to the above-mentioned modified example in which both functions are combined.

[0019] The tracking unit 118 (tracking means) tracks the subject detected by the object detection unit 116 by inferring to which position the subject has moved, and acquires center position information and size information of the subject (hereinafter referred to as "subject to be tracked") as a tracking result. The tracking unit 118 is realized, for example, by using DL (Deep Learning) using machine learning or template matching without machine learning. Note that the method of tracking processing by the tracking unit 118 is not limited to the above-mentioned examples, and existing technologies can be used. The image post-processing unit 119 converts, for example, the RGB signal or YUV signal recorded in the memory 115 into a display image to be displayed on the display unit 106, and outputs the converted display image to the display frame superimposition unit 123. Furthermore, the image post-processing unit 119 converts the RGB signal or YUV signal recorded in the memory 115 into a recording image to be written to the recording medium 107, and outputs the converted recording image to the recording unit 122. Incidentally, for example, an SD memory card or a flash memory such as Compact Flash (registered trademark) is used as the recording medium 107. Also, the image for recording may be, for example, compressed data to which JPEG compression is applied.

[0020] The temperature recording determination unit 120 (recording determination means) determines whether or not to record temperature information indicating the temperature of the subject. This determination will be described in detail later. The temperature acquisition unit 121 (acquisition and recording means) acquires temperature information of the subject to be recorded from the area of ​​the subject (hereinafter, sometimes referred to as the "subject to be recorded") for which the temperature recording determination unit 120 has determined that the temperature information should be recorded in the invisible light image. Such acquisition is performed based on the center position information and area information included in the detection result of the subject to be recorded. The temperature information may be acquired as a thermal image or a temperature distribution map, or may be acquired as Exif information. The recording unit 122 (acquisition and recording means) writes the temperature information acquired by the temperature acquisition unit 121 to the detection result and identification result of the subject to be recorded in the recording medium 107. As a result, the temperature information of the subject is recorded in the recording medium 107 in association with at least the center position information and area information (i.e., the above-mentioned image information) included in the detection result of the subject. Therefore, the temperature information of the subject recorded on the recording medium 107 can be used in the thermal sensation presentation device together with the central position information and area information of the subject that are linked to it and recorded on the recording medium 107.

[0021] The display frame superimposing unit 123 generates a selection frame from the center position and size of the subject detected by the object detection unit 116, and outputs a display image in which the generated selection frame is superimposed on the subject to the display unit 106. Such generation and superimposition of the selection frame is performed based on center position information and size information of the subject detected by the object detection unit 116. The selection frame will be described later. Also, the display frame superimposing unit 123 generates a tracking frame from the center position and size of the subject to be tracked, and outputs a display image in which the generated tracking frame is superimposed on the subject to be tracked to the display unit 106. Such generation and superimposition of the tracking frame is performed based on center position information and size information of the subject to be tracked. Also, the display frame superimposing unit 123 generates a temperature recording frame from the center position and size of the subject to be recorded, and outputs a display image in which the generated temperature recording frame is superimposed on the subject to be recorded to the display unit 106. Such generation and superimposition of the temperature recording frame is performed based on center position information and size information of the subject to be recorded.

[0022] The display unit 106 is composed of, for example, an LCD panel or an organic EL panel, and displays the display image output by the display frame overlapping unit 123. The display unit 106 also displays images recorded in a system memory 113 (described later) and images recorded in the recording unit 122. The display unit 106 also displays an operation user interface for receiving instructions from a user. The display unit 106 may have multiple display devices, such as an EVF (electronic viewfinder) and a rear monitor provided on the photographer's side (rear). The display unit 106 may be capable of simultaneously outputting to multiple display devices, or may be configured to selectively display by switching between them.

[0023] FIG. 2A is a diagram showing the display unit 106 displaying a display image on which a rectangular temperature recording frame 201 (first display information) is superimposed. In the case of FIG. 2A, of the three subjects 202 to 204 displayed in the display image of the display unit 106, the subject 202 is a subject to be recorded. Therefore, the display frame superimposing unit 123 (first display control means) generates the temperature recording frame 201 based on the detection result and the identification result of the subject 202, and outputs the display image in which the generated temperature recording frame 201 is superimposed on the subject 202 to the display unit 106. This allows the user to visually recognize from the temperature recording frame 201 that the subject 202 is a subject to be recorded, that is, a subject whose temperature information is to be recorded on the recording medium 107 by the recording unit 122. In this way, the subject 202, which is a subject to be recorded, is highlighted by the temperature recording frame 201, improving the user's visibility.

[0024] FIG. 2B is a diagram showing the display unit 106 displaying a display image with a temperature recording icon 205 (first display information) superimposed thereon. The temperature recording icon 205 is a simple design of a glass thermometer. In the case of FIG. 2B, as in the case of FIG. 2A, of the three subjects 202 to 204 displayed in the display image of the display unit 106, the subject 202 is the subject to be recorded. Therefore, the display frame superimposing unit 123 (first display control means) generates the temperature recording icon 205 based on the detection result and the identification result of the subject 202, and outputs to the display unit 106 a display image in which the generated temperature recording icon 205 is superimposed near the subject 202. This allows the user to visually recognize from the temperature recording icon 205 that the subject 202 is the subject to be recorded, that is, the subject whose temperature information is to be recorded on the recording medium 107 by the recording unit 122. In this way, the subject 202, which is the subject to be recorded, is highlighted with the temperature recording icon 205, improving visibility for the user.

[0025] Note that the highlighting of the recording target subject is not limited to the temperature recording frame 201 or the temperature recording icon 205. For example, the recording target subject may be highlighted with characters, symbols, other simple figures or patterns, or effect processing, etc. This also applies to the above-mentioned selection frame and tracking frame.

[0026] Returning to FIG. 1, the system control unit 108 controls the entire imaging device 100. The system control unit 108 realizes each process by executing a program recorded in the non-volatile memory 112. The release button 109, the mode changeover switch 110, and the operation unit 111 input various operation instructions to the system control unit 108 according to user operations. The release button 109 inputs to the system control unit 108 a state in which the release button 109 is being operated, that is, a so-called half-pressed state (instruction to prepare for shooting), and a state in which the release button 109 is fully pressed (instruction to shoot), that is, the operation of the release button 109 is completed. The system control unit 108 controls the entire system so as to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, and AWB (auto white balance) processing in the above-mentioned half-pressed state. In addition, the system control unit 108 controls the entire system so as to start a series of operations of shooting processing from reading out signals from the visible light imaging element 103 and the non-visible light imaging element 104 to writing an image for recording to the recording medium 107 in the above-mentioned fully pressed state.

[0027] The mode changeover switch 110 changes the operation mode of the system control unit 108 to one of a still image recording mode, a video shooting mode, a playback mode, and the like. In the still image recording mode and the video shooting mode, the shutter speed of the shutter 102 can be set by a shutter speed setting. In addition, the still image recording mode and the video shooting mode have a temperature recording setting mode. In the temperature recording setting mode, the recording of temperature information can be switched ON / OFF, and the subject for which the temperature information is recorded can be switched by a priority type setting or a subject recognition setting. The user can switch to one of these modes included in the menu by operating the mode changeover switch 110.

[0028] The operation unit 111 (operation means) has a plurality of operation members. With each operation member, the user can select and operate various function icons displayed on the display unit 106, and can intuitively perform various settings by operating hardware such as various function buttons and four-way key sticks (up, down, left, right, etc.). In other words, the user can perform touch operation, button operation, key operation, and stick operation by using each operation member of the operation unit 111. The non-volatile memory 112 is an electrically erasable and recordable memory. For example, an EEPROM is used for the non-volatile memory 112, and constants and programs for the operation of the system control unit 108 are stored.

[0029] The system memory 113 uses a RAM, and stores constants and variables for the operation of the system control unit 108, programs read from the nonvolatile memory 112, and the like. The system memory 113 is also used for temporarily recording data generated by the operation of each block of the imaging device 100.

[0030] Next, the temperature recording control in the first embodiment will be described in detail. Fig. 3 is a flowchart showing the temperature recording control in the first embodiment. The flowchart in Fig. 3 (a control method for an imaging device) is realized by the system control unit 108 (computer) reading a program stored in the non-volatile memory 112 into the system memory 113, executing it, and controlling each component of the image processing unit 105. The flowchart in Fig. 3 is started when the operation mode of the system control unit 108 is set to the temperature recording setting mode, and further, the recording of temperature information is ON. These points are the same for the flowcharts showing the temperature recording control in each of the second, third, and fourth embodiments described later.

[0031] In the temperature recording control in the first embodiment, temperature information is recorded based on the detection result of the object detection unit 116 and the identification result of the object identification unit 117. FIG. 4 is a diagram showing a table in which the detection result of the object detection unit 116, the identification result of the object identification unit 117, and the judgment result of the temperature recording judgment unit 120 correspond to five scenes assumed in the first embodiment. In the table of FIG. 4, the detection result "x" indicates that the object detection unit 116 was unable to detect the subject, and the detection result "o" indicates that the object detection unit 116 was able to detect the subject. The identification result "no identification" indicates that the object identification unit 117 was unable to identify the type of the subject detected by the object detection unit 116. In addition, the identification results "person", "motorcycle", and "dog" indicate that the object identification unit 117 identified the type of the subject detected by the object detection unit 116 as a person, a motorcycle, and a dog.

[0032] The judgment result "x" indicates that the temperature recording judgment unit 120 judges not to record the temperature information of the subject, and the judgment result "o" indicates that the temperature recording judgment unit 120 judges that the temperature information of the subject is to be recorded. In the first embodiment, the temperature recording judgment unit 120 judges that the temperature information of the subject is to be recorded when the type of the subject is a living organism. Therefore, in scenes 3 and 5, the types of the subject are a person and a dog, and the temperature recording judgment unit 120 judges that the temperature information of the subject is to be recorded, so the judgment result is "o". Note that the types of living organisms are not limited to the people and dogs listed in the table of FIG. 4, but may include organisms other than people and dogs (cats, birds, etc.), and may also include parts of the organism such as the head and torso.

[0033] On the other hand, when the object identification unit 117 is unable to identify the type or when the type of the subject is a non-living object, the temperature record determination unit 120 determines not to record the temperature information of the subject. Therefore, in scene 2, the object identification unit 117 is unable to identify the type and the temperature record determination unit 120 determines not to record the temperature information of the subject, so the determination result is "X". Also, in scene 4, the type of the subject is a motorcycle and the temperature record determination unit 120 determines not to record the temperature information of the subject, so the determination result is "X". Note that the type of non-living object is not limited to the motorcycles listed in the table of FIG. 4, and may include vehicles other than motorcycles (bicycles, cars, trains, airplanes, ships, etc.). Incidentally, when the object detection unit 116 is unable to detect the subject, as in scene 1, the object identification unit 117 does not identify the subject and the temperature record determination unit 120 does not determine the subject.

[0034] Hereinafter, the temperature recording control in the first embodiment will be described with reference to the flowchart in Fig. 3. In step S300, the visible light imaging element 103 acquires a visible light image (first imaging step). Furthermore, the image pre-processing unit 114 converts the visible light image into an RGB signal or a YUV signal, and writes it to the memory 115. Furthermore, the non-visible light imaging element 104 acquires a non-visible light image (second imaging step). Furthermore, the image pre-processing unit 114 writes the non-visible light image to the memory 115.

[0035] In step S301, the object detection unit 116 detects one or more objects from the RGB signal or YUV signal recorded in the memory 115. Furthermore, when the object detection unit 116 detects one or more objects, it acquires the above-mentioned number information, size information, center position information, area information, etc. for the detected one or more objects as detection results (detection process). In step S302, the system control unit 108 determines whether the object detection unit 116 has detected the object. If the system control unit 108 determines that the object detection unit 116 has detected the object, the process proceeds to step S303. On the other hand, if the system control unit 108 determines that the object detection unit 116 has not detected the object, the temperature recording control shown in the flowchart of FIG. 3 ends.

[0036] In step S303, the object identification unit 117 identifies the object detected by the object detection unit 116, thereby acquiring the above-mentioned class information as an identification result (identification step). In step S304, the temperature record determination unit 120 determines whether or not to record the temperature information of the object detected by the object detection unit 116 based on the identification result (recording determination step). If the type of the identification result of the object identification unit 117 is a living organism such as a person or a dog, the temperature record determination unit 120 determines to record the temperature information of the object, and the process proceeds to step S305. On the other hand, if the type of the identification result of the object identification unit 117 is a non-living organism such as a motorcycle, or if the object identification unit 117 is unable to identify the object, the temperature record determination unit 120 determines not to record the temperature information of the object. In this way, if the temperature record determination unit 120 determines not to record the temperature information of the object, the temperature record control shown in the flowchart of FIG. 3 ends.

[0037] In step S305, temperature acquisition unit 121 acquires temperature information of the subject that has been determined by temperature recording determination unit 120, that is, temperature information of the recording target subject. Furthermore, recording unit 122 links the temperature information of the recording target subject acquired by temperature acquisition unit 121 to the detection result and identification result of the recording target subject and writes it to recording medium 107 (acquisition and recording step). Thereafter, the temperature recording control shown in the flowchart of Fig. 3 ends. Note that if object detection unit 116 detects multiple subjects, steps S303 to S305 are performed for each subject detected by object detection unit 116.

[0038] As described above, the imaging device 100 according to the first embodiment records the temperature information of the subject in the recording medium 107 in response to a determination based on the detection result of the object detection unit 116 and the identification result of the object identification unit 117. This allows the imaging device 100 to record temperature information about an appropriately narrowed-down subject. Furthermore, by using the temperature information and the like recorded in the recording medium 107 of the imaging device 100, the thermal sensation presentation device can limit the subject presenting the thermal sensation to a living body or the like without performing high-spec post-processing such as detection required to limit the subject.

[0039] The present invention is not limited to the first embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. For example, contrary to the first embodiment, the imaging device 100 may not record the temperature information of the subject when the type of the subject is identified as a living organism, and may record the temperature information of the subject when the type of the subject is identified as a non-living organism. Furthermore, the imaging device 100 may record the temperature information of the subject according to a type different from living organism / non-living organism.

[0040] <Second embodiment> The second embodiment will be described below with reference to Figs. 5 and 6. Here, the differences from the first embodiment will be mainly described. Fig. 5 is a flowchart showing temperature recording control in the second embodiment. In the temperature recording control in the second embodiment, temperature information is recorded based on the detection result of the object detection unit 116 and the identification result of the object identification unit 117, as well as on a selection by the user. The temperature recording control in the second embodiment will be described below with reference to the flowchart in Fig. 5. Steps S500 to S503 are similar to steps S300 to S303 described above, and therefore their description will be omitted.

[0041] In step S504, the display frame superimposing unit 123 generates a selection frame for the subject detected by the object detection unit 116 from its center position and size, and outputs to the display unit 106 a display image in which the generated selection frame is superimposed on the subject. As a result, the display unit 106 displays the display image in which the selection frame is superimposed on the subject. However, the display frame superimposing unit 123 (second display control means) generates and superimposes a selection frame (second display information) only for subjects whose type is a living organism such as a person or a dog as a result of the identification by the object identification unit 117 in step S503. Note that if the object detection unit 116 detects multiple subjects, steps S503 and S504 are performed for each subject detected by the object detection unit 116.

[0042] 6 is a diagram showing display unit 106 displaying a display image on which selection frames 601-603 are superimposed. In the case of FIG. 6, four subjects 604-607 are displayed in the display image on display unit 106. The type of subject 604 is a bird, which is a living organism type. The types of subjects 605 and 606 are a dog, which is a living organism type. The type of subject 607 is a motorcycle, which is a non-living organism type. Therefore, display frame superimposing unit 123 generates selection frames 601-603 based on the detection results and identification results of subjects 604-606, and outputs a display image in which the generated selection frames 601-603 are superimposed on subjects 604-606 to display unit 106.

[0043] Returning to Fig. 5, in step S505, the temperature record determination unit 120 determines whether or not to record temperature information of a subject on which a selection frame is superimposed in the display image displayed on the display unit 106, based on a user's selection. If the user selects a subject on which a selection frame is superimposed in the display image displayed on the display unit 106, the temperature record determination unit 120 determines that the temperature information of the subject is to be recorded, and the process proceeds to step S506. On the other hand, if the user does not select a subject on which a selection frame is superimposed in the display image displayed on the display unit 106, the temperature record determination unit 120 determines that the temperature information of the subject is not to be recorded, and the temperature record control shown in the flowchart of Fig. 5 ends.

[0044] The user selects the subject by using the operation unit 111 through a touch operation, a button operation, a key operation, or a stick operation. In addition, the system control unit 108 allows the user to select the subject on which a selection frame is superimposed in the display image displayed on the display unit 106, and prevents the user from selecting the subject on which a selection frame is not superimposed. Specifically, in the display image of the display unit 106 shown in FIG. 6, the user can select the subjects 604 to 606 on which selection frames 601 to 603 are superimposed, and prevents the user from selecting the subject 607 on which a selection frame is not superimposed. Then, when the user selects at least one of the subjects 604 to 606 on which the selection frames 601 to 603 are superimposed, the temperature recording determination unit 120 determines that the temperature information of the subject is to be recorded. When selecting a subject, the user can visually recognize from the selection frames 601 to 603 that the subject is selectable by the operation unit 111. That is, subjects 604 to 606 that the user can select using operation unit 111 are highlighted by selection frames 601 to 603, improving visibility for the user.

[0045] Returning to Fig. 5, step S506 is the same as step S305 described above, and therefore its description will be omitted. After step S506, the temperature recording control shown in the flowchart of Fig. 5 ends. Note that, if the user selects multiple subjects with the operation unit 111, step S506 is performed for each subject selected by the user.

[0046] From the above, in the imaging device 100 according to the second embodiment, the user can select a subject for which temperature information is to be recorded in the recording medium 107 from among one or more subjects displayed in the display image of the display unit 106. However, the imaging device 100 limits the subjects that the user can select by type based on the identification result of the object identification unit 117. In this way, the imaging device 100 records the temperature information of the subject in the recording medium 107 in accordance with the detection result of the object detection unit 116 and the identification result of the object identification unit 117, as well as the determination based on the selection by the user. This allows the imaging device 100 to record temperature information about the appropriately narrowed down subjects. In addition, the thermal sensation presentation device can limit the subjects that present the thermal sensation to those selected by the user in the imaging device 100 by using the temperature information recorded in the recording medium 107 of the imaging device 100, without performing high-spec post-processing required for limiting the subjects.

[0047] The present invention is not limited to the second embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. For example, unlike the second embodiment, the imaging device 100 may allow the user to select any of one or more subjects to be displayed in the display image of the display unit 106. In this case, the identification result of the object identification unit 117 and the generation and superimposition of a selection frame by the display frame superimposition unit 123 are not necessary. The imaging device 100 may also allow the user to select an area in the display image to be displayed on the display unit 106, the area of ​​which temperature information is to be recorded in the recording medium 107. In this case, the imaging device 100 records the temperature information of the area selected by the user in association with information identifying the area, etc., in the recording medium 107. As a result, the imaging device 100 can record the temperature information of the area not detected by the object detection unit 116 and the temperature information of the area not identified by the object identification unit 117 in the recording medium 107 by selecting each area by the user.

[0048] <Third embodiment> Hereinafter, the third embodiment will be described with reference to FIG. 7 and FIG. 8. Here, the differences from the first embodiment will be mainly described. FIG. 7 is a flowchart showing temperature recording control in the third embodiment. In the temperature recording control in the third embodiment, temperature information is recorded based on the detection result of the object detection unit 116 and the identification result of the object identification unit 117, as well as on the priority type setting. The priority type setting is performed by a touch operation, a button operation, a key operation, or a stick operation performed by the user using the operation unit 111. Also, the priority type setting may be switched in advance or at any time.

[0049] FIG. 8 is a diagram showing a list of priority type settings assumed in the third embodiment. In the list of FIG. 8, "OFF" indicates that priority type setting has not been performed, and "person priority" indicates that the type of subject for which priority type setting is set is a person. Furthermore, "dog priority" indicates that the type of subject for which priority type setting is set is a dog, and "bird priority" indicates that the type of subject for which priority type setting is set is a bird. Note that the types of subjects for which priority type setting is set are not limited to people, dogs, and birds listed in the list of FIG. 8, but include types of living organisms other than people, dogs, and birds. Furthermore, the types of subjects for which priority type setting is set may include parts of living organisms such as heads and torsos, or may include non-living organisms such as cars and motorcycles.

[0050] Hereinafter, the temperature recording control in the third embodiment will be described with reference to the flowchart in FIG. 7. Steps S700 to S703 are similar to steps S300 to S303 described above, and therefore their description will be omitted. In step S704, the temperature recording determination unit 120 determines whether or not to record temperature information of the subject detected by the object detection unit 116, based on the identification result and a priority type setting that is performed in advance or on an as-needed basis. If the type of the identification result of the object identification unit 117 matches the type of the subject that is set as the priority type, the temperature recording determination unit 120 determines that the temperature information of the subject is to be recorded, and the process proceeds to step S705.

[0051] On the other hand, if the type of the identification result of the object identification unit 117 does not match the type of the object set as the priority type, if the object identification unit 117 is unable to identify the object, or if the priority type is not set, the temperature recording determination unit 120 determines not to record the temperature information of the object. In this way, when the temperature recording determination unit 120 determines not to record the temperature information of the object, the temperature recording control shown in the flowchart of FIG. 7 ends. Step S705 is the same as step S305 described above, and therefore a description thereof will be omitted. After step S705, the temperature recording control shown in the flowchart of FIG. 7 ends. Note that, if the object detection unit 116 detects multiple objects, steps S703 to S705 are performed for each object detected by the object detection unit 116.

[0052] As described above, the imaging device 100 according to the third embodiment records the temperature information of the subject in the recording medium 107 in accordance with the detection result of the object detection unit 116, the identification result of the object identification unit 117, and the determination based on the priority type setting. This allows the imaging device 100 to record temperature information for appropriately narrowed-down subjects. In addition, the thermal sensation presentation device can limit the subject presenting the thermal sensation to the subject set as the priority type by the user in the imaging device 100 by using the temperature information recorded in the recording medium 107 of the imaging device 100, without performing high-spec post-processing required for limiting the subject. Note that the present invention is not limited to the third embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. For example, unlike the third embodiment, the imaging device 100 may allow the user to set multiple types as priority types.

[0053] <Fourth embodiment> The fourth embodiment will be described below with reference to FIG. 9. Here, the differences from the fourth embodiment will be mainly described. FIG. 9 is a flowchart showing temperature recording control in the fourth embodiment. In the temperature recording control in the fourth embodiment, temperature information is recorded based on the detection result of the object detection unit 116 and the authentication result of the object identification unit 117. The authentication result of the object identification unit 117 is a result of the object identification unit 117 confirming, by its authentication function, whether the subject detected by the object detection unit 116 is the same as a specific subject registered in advance. In this way, the object identification unit 117 acquires the authentication result as the above-mentioned identification result. The user can register a specific subject in advance by subject authentication setting (authentication setting). In the subject authentication setting, a specific person or a specific pet is registered in advance as a specific subject. The subject authentication setting is performed by a touch operation, a button operation, a key operation, or a stick operation performed by the user using the operation unit 111.

[0054] Hereinafter, the temperature recording control in the fourth embodiment will be described with reference to the flowchart in Fig. 9. Steps S900 to S902 are similar to steps S300 to S302 described above, and therefore their description will be omitted. In step S903, the object identification unit 117 acquires an authentication result for the subject detected by the object detection unit 116. In step S904, the temperature recording determination unit 120 determines whether or not to record the temperature information of the subject detected by the object detection unit 116 based on the authentication result.

[0055] If the authentication result of the object identification unit 117 is "the subject detected by the object detection unit 116 is the same as the specific subject registered in advance in the subject recognition setting", the temperature recording determination unit 120 determines to record the temperature information of the subject, and the process proceeds to step S905. On the other hand, if the authentication result of the object identification unit 117 is "the subject detected by the object detection unit 116 is not the same as the specific subject registered in advance in the subject recognition setting", the temperature recording determination unit 120 determines not to record the temperature information of the subject. In this way, when the temperature recording determination unit 120 determines not to record the temperature information of the subject, the temperature recording control shown in the flowchart of FIG. 9 ends.

[0056] In step S905, temperature acquisition unit 121 acquires temperature information of the subject that is the subject of judgment by temperature recording judgment unit 120, that is, temperature information of the recording target subject. Furthermore, recording unit 122 writes the temperature information of the recording target subject acquired by temperature acquisition unit 121 to the detection result and authentication result of the recording target subject in recording medium 107. Thereafter, the temperature recording control shown in the flowchart of Fig. 7 ends. Note that, if object detection unit 116 detects multiple subjects, steps S903 to S905 are performed for each subject detected by object detection unit 116.

[0057] As described above, the imaging device 100 according to the fourth embodiment records the temperature information of the subject in the recording medium 107 in response to a determination based on the detection result of the object detection unit 116 and the authentication result, which is the identification result of the object identification unit 117. This allows the imaging device 100 to record temperature information for an appropriately narrowed-down subject. In addition, the thermal sensation presentation device can limit the subject presenting the thermal sensation to the subject set by the user for subject recognition in the imaging device 100 by using the temperature information recorded in the recording medium 107 of the imaging device 100 without performing high-spec post-processing required for limiting the subject. Note that the present invention is not limited to the fourth embodiment, and various modifications and changes are possible within the scope of the gist of the present invention. For example, unlike the fourth embodiment, the imaging device 100 may allow the user to set the subject recognition in real time.

[0058] <Thermal sensation presentation device> The first to third warm / cold sensation presentation devices will be described below with reference to Fig. 10 to Fig. 12. The warm / cold sensation presentation devices described here are capable of presenting a warm / cold sensation based on image information from the imaging device 100 according to the first to fourth embodiments and temperature information linked to the image information.

[0059] FIG. 10 is a conceptual diagram showing a state in which the first warm / cold sense presentation device 1000 is used. As shown in FIG. 10, the first warm / cold sense presentation device 1000 has a screen 1001 and a projector 1002 arranged at a distance from the screen 1001. The screen 1001 may be omitted from the configuration of the first warm / cold sense presentation device 1000. In this case, the screen 1001 is prepared separately from the first warm / cold sense presentation device 1000. The projector 1002 irradiates, i.e. projects, an image 1003 obtained from input image data onto the screen 1001. As a result, the image 1003 is displayed on the screen 1001.

[0060] The image 1003 includes a person 1004 and a tree 1005. The screen 1001 has a plurality of Peltier elements 1006 arranged in a matrix on the back side. The position of the screen 1001 is adjusted relative to the projection angle of the projector 1002. The part of the screen 1001 overlapping with the person 1004 becomes a temperature control target and is temperature-adjusted because the imaging device 100 according to each embodiment has acquired temperature information linked to the image information. While visually recognizing the person 1004 projected on the screen 1001, the user can feel the temperature corresponding to the person 1004 by touching the part of the screen 1001 corresponding to the person 1004.

[0061] FIG. 11 is an image diagram showing a state in which the second warm / cold sense presentation device 1100 is used. As shown in FIG. 11, the second warm / cold sense presentation device 1100 has a warm / cold sense presentation unit 1101 and a printing unit 1102. The printing unit 1102 may be omitted from the configuration of the second warm / cold sense presentation device 1100. In this case, the printing unit 1102 is prepared separately from the second warm / cold sense presentation device 1100. The printing unit 1102 is preferably connected to the second warm / cold sense presentation device 1100 so as to be capable of communicating with the second warm / cold sense presentation device 1100. The printing unit 1102 outputs (prints out) a printed matter 1105 in which an image 1103 obtained from input image data is printed on a sheet 1104. As a result, the image 1103 is displayed on the sheet 1104. The image 1103 includes a human 1106 and a tree 1107.

[0062] The hot and cold sense presentation unit 1101 has a plurality of Peltier elements 1108 arranged in a matrix on the front side. Note that the hot and cold sense presentation unit 1101 may have each Peltier element 1108 in a convex shape. The second hot and cold sense presentation device 1100 is used by overlapping a printed matter 1105 on the hot and cold sense presentation unit 1101. The hot and cold sense presentation unit 1101 preferably has a detection unit that detects that the printed matter 1105 is overlapped, and an alignment unit that aligns the printed matter 1105 with the printed matter 1105. The part of the hot and cold sense presentation unit 1101 that overlaps with the person 1106 is subject to temperature control and is temperature-adjusted because the image capture device 100 according to each embodiment acquires temperature information linked with image information. The user can feel the temperature corresponding to the person 1106 by touching the person 1106 on the printed matter 1105 while visually recognizing the person 1106 on the printed matter 1105.

[0063] FIG. 12 is an image diagram showing a usage state of the third warm / cold sense presentation device 1200. As shown in FIG. 12, the third warm / cold sense presentation device 1200 has a warm / cold sense presentation unit 1201 on the back side of an image display unit such as a liquid crystal display. The warm / cold sense presentation unit 1201 has a warm / cold sense presentation area 1202 that presents a warm / cold sense. In the warm / cold sense presentation area 1202, a plurality of warm / cold sense presentation elements 1203, for example, composed of Peltier elements, are arranged in a matrix (seven rows vertically and eight columns horizontally in the configuration shown in FIG. 12). On the other hand, input image data 1204 input to the image data input unit of the third warm / cold sense presentation device 1200 is data of an image including a person 1205 and a tree 1206. In the input image data 1204, a rectangular person area 1207 inscribed by the person 1205 is acquired. In addition, the warm / cold sensation presentation unit 1201 can scale the entire angle of view of the input image data 1204 to the warm / cold sensation presentation area 1202 .

[0064] At this time, in the hot / cold sensation presentation area 1202, a person-corresponding area 1208 is formed at a position corresponding to the person area 1207, with the same aspect ratio as that of the person area 1207 being maintained. Then, a person hot / cold sensation presentation area 1209 is set, which is composed of the hot / cold sensation presentation elements 1203 and overlaps at least a part of the person-corresponding area 1208. The person hot / cold sensation presentation area 1209 corresponds to an object-occupied area occupied by an object in an image, and whether or not to execute temperature control, i.e., whether or not to set the area as a target area for temperature control, is determined by temperature information linked together with image information by the imaging device 100 according to each embodiment. If temperature information corresponding to the person 1205 is acquired, the person hot / cold sensation presentation area 1209 presents the temperature, and the user can feel the temperature (body temperature) of the person 1205 by touching a part of the person hot / cold sensation presentation area 1209 on the image display unit with a finger 1210.

[0065] <Other> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the imaging device 100 according to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. For example, the temperature acquisition unit 121 and the recording unit 122 may acquire and average a plurality of pieces of temperature information of the subject that is the subject of judgment by the temperature record judgment unit 120, that is, the temperature information of the subject to be recorded, and record the averaged temperature information of the subject to be recorded on the recording medium 107. This allows the imaging device 100 to record highly accurate temperature information on the recording medium 107.

[0066] Specifically, the temperature acquisition unit 121 acquires multiple pieces of temperature information of the subject to be recorded during the open time of the shutter 102 (shutter open time), that is, during the exposure time of the visible light imaging element 103 and the non-visible light imaging element 104. The temperature acquisition unit 121 acquires more temperature information of the subject to be recorded the longer the open time of the shutter 102, that is, the longer the exposure time of the visible light imaging element 103 and the non-visible light imaging element 104. In other words, the temperature acquisition unit 121 acquires multiple pieces of temperature information of the subject that has been determined by the temperature record determination unit 120 according to the shutter speed or exposure time of the imaging device 100.

[0067] Furthermore, when the opening time of the shutter 102 is relatively short (the shutter speed is fast), that is, when the exposure time of the visible light imaging element 103 and the non-visible light imaging element 104 is relatively short, the temperature acquisition unit 121 may acquire, for example, temperature information for a captured frame and frames before and after that. That is, the temperature acquisition unit 121 acquires multiple pieces of temperature information of the subject that is the subject of judgment by the temperature recording and judgment unit 120, before and after the opening time of the shutter 102 or before and after the exposure time of the visible light imaging element 103 and the non-visible light imaging element 104.

[0068] Furthermore, when the tracking unit 118 can track the same subject over a plurality of frames, that is, while the tracking unit 118 is tracking, the temperature acquisition unit 121 and the recording unit 122 may continuously acquire and average temperature information of the same subject being tracked by the tracking unit 118. Furthermore, when the tracking unit 118 loses sight of the subject being tracked while tracking by the tracking unit 118, the temperature acquisition unit 121 may erase the temperature information of the subject being tracked that was continuously acquired before the tracking unit 118 lost sight of the subject being tracked. This enables the imaging device 100 to remove, for example, temperature information that is unnecessary for the thermal sensation presentation device.

[0069] The present invention can also be realized by supplying a program that realizes one or more functions of each of the above-mentioned embodiments to a system or device via a network or a storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0070] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A first imaging means for acquiring a visible light image; A second imaging means for acquiring a non-visible light image; a detection means for detecting one or more objects based on the visible light image to obtain one or more detection results; a classification means for acquiring one or more classification results by classifying one or more types of subjects detected by the detection means based on the visible light image; a record determination means for determining whether or not to record temperature information of the subject based on the detection result and the identification result; and an acquisition and recording means for, when the recording determination means determines that the temperature information of the subject should be recorded, acquiring the temperature information of the subject that is the subject of the determination by the recording determination means based on the non-visible light image and the detection result, and recording the information on a recording medium. (Configuration 2) The imaging device according to configuration 1, wherein the identification means acquires the identification result by using machine learning. (Configuration 3) An imaging device as described in Configuration 1 or 2, characterized in that it further comprises a first display control means for superimposing, on a display image displayed on a display unit based on the visible light image, first display information indicating that the subject is one whose temperature information is recorded on the recording medium by the acquisition and recording means, based on the detection result and the identification result. (Configuration 4) A manipulation means is provided for allowing a user to select a subject in a display image displayed on a display unit based on the visible light image, The imaging device according to any one of configurations 1 to 3, characterized in that the recording determination means determines whether to record temperature information of the subject based on a user's selection via the operation means in addition to the detection result and the identification result. (Configuration 5) The imaging device described in Configuration 4 is characterized in that it is provided with a second display control means for superimposing, on the display image, second display information indicating that the subject is one that the user can select using the operation means, based on the detection result and the identification result. (Configuration 6) The camera includes an operation means for a user to set a priority type of subject for which temperature information is to be recorded with priority, The imaging device according to any one of configurations 1 to 3, wherein the recording determination means determines whether or not to record temperature information of the subject based on the priority type setting in addition to the detection result and the identification result. (Configuration 7) The camera includes an operation unit for a user to perform authentication setting for registering in advance a subject whose temperature information is to be recorded, The imaging device according to any one of configurations 1 to 3, wherein the identification means obtains, as the identification result, an authentication result indicating whether or not one or more subjects detected by the detection means are identical to subjects pre-registered in the authentication setting. (Configuration 8) The imaging device according to any one of configurations 4 to 7, wherein the operation means allows touch operation, button operation, key operation, or stick operation. (Configuration 9) The imaging device according to any one of configurations 1 to 8, further comprising a detection and identification means serving as both the detection means and the identification means. (Configuration 10) A first imaging means for acquiring a visible light image; A second imaging means for acquiring a non-visible light image; a detection means for detecting one or more objects based on the visible light image to obtain one or more detection results; an operation means for allowing a user to select a subject in a display image displayed on a display unit based on the visible light image; a record determination means for determining whether or not to record temperature information of a subject based on a selection made by a user using the operation means; and an acquisition and recording means for, when the recording determination means determines that the temperature information of the subject should be recorded, acquiring the temperature information of the subject that is the subject of the determination by the recording determination means based on the non-visible light image and the detection result, and recording the information on a recording medium. (Configuration 11) The imaging device according to configuration 10, wherein the operation means allows touch operation, button operation, key operation, or stick operation. (Configuration 12) An imaging device as described in Configuration 10 or 11, characterized in that it is provided with a first display control means for superimposing, on the display image, first display information indicating that the subject is one whose temperature information is recorded on the recording medium by the acquisition and recording means, based on the detection result and selection by the user using the operation means. (Configuration 13) An imaging device described in any one of configurations 1 to 12, characterized in that the acquisition and recording means links central position information and area information of the subject contained in the detection result with temperature information acquired from the non-visible light image based on the central position information and the area information and records them on the recording medium. (Configuration 14) The imaging device according to configuration 13, wherein the detection means obtains size information of the subject, the central position information, and the area information as the detection result by using machine learning. (Configuration 15) An imaging device described in any one of configurations 1 to 14, characterized in that the acquisition and recording means acquires and averages multiple pieces of temperature information of the subject that is the subject of judgment by the recording and judgment means, and records the averaged temperature information of the subject on the recording medium. (Configuration 16) The imaging device according to configuration 15, wherein the acquisition and recording means acquires multiple pieces of temperature information of the subject that has been determined by the recording and determination means in accordance with a shutter speed or exposure time of the imaging device. (Configuration 17) The imaging device described in Configuration 15, wherein the acquisition and recording means acquires multiple pieces of temperature information of the subject that has been judged by the recording and judgment means, over a period before and after the shutter release time or before and after the exposure time of the imaging device. (Configuration 18) A tracking unit is provided for tracking a subject that is the subject of the recording determination unit, The imaging device according to any one of configurations 1 to 17, wherein the acquisition and recording means continuously acquires and averages temperature information of the subject being tracked by the tracking means while the tracking means is tracking, and records the averaged temperature information of the subject on the recording medium. (Configuration 19) The imaging device described in Configuration 18, wherein the acquisition and recording means, when the tracking means loses sight of the subject being tracked while tracking the subject, erases the temperature information of the subject being tracked that was continuously acquired by the tracking means before the tracking means lost sight of the subject being tracked. (Method 1) A first imaging step of acquiring a visible light image; A second imaging step of acquiring a non-visible light image; a detection step of detecting one or more objects based on the visible light image to obtain one or more detection results; a classification step of acquiring one or more classification results by classifying the one or more types of subjects detected in the detection step based on the visible light image; a recording determination step of determining whether or not to record temperature information of the subject based on the detection result and the identification result; and an acquisition and recording step of, when the recording determination step determines that the temperature information of the subject is to be recorded, acquiring temperature information of the subject determined in the recording determination step based on the non-visible light image and the detection result, and recording the temperature information on a recording medium. (Program 1) A program for causing a computer to execute each means of the imaging device according to any one of configurations 1 to 19. [Explanation of symbols]

[0071] 100 Imaging device 103 Visible light imaging element (first imaging means) 104 Non-visible light imaging element (second imaging means) 107 Recording media 116 Object detection unit (detection means) 117 Object identification unit (identification means) 120 Temperature recording and judgment unit (recording and judgment means) 121 Temperature acquisition unit (acquisition and recording means) 122 Recording unit (acquisition and recording means)

Claims

1. A first imaging means for acquiring a visible light image; A second imaging means for acquiring a non-visible light image; a detection means for detecting one or more objects based on the visible light image to obtain one or more detection results; a classification means for acquiring one or more classification results by classifying one or more types of subjects detected by the detection means based on the visible light image; a record determination means for determining whether or not to record temperature information of the subject based on the detection result and the identification result; and an acquisition and recording means for, when the recording determination means determines that the temperature information of the subject should be recorded, acquiring the temperature information of the subject that is the subject of the determination by the recording determination means based on the non-visible light image and the detection result, and recording the information on a recording medium.

2. The imaging apparatus according to claim 1 , wherein the identification unit acquires the identification result by using machine learning.

3. 2. The imaging device according to claim 1, further comprising a first display control means for superimposing, on a display image displayed on a display unit based on the visible light image, first display information indicating that the subject is one about which temperature information is to be recorded on the recording medium by the acquisition and recording means, based on the detection result and the identification result.

4. an operation means for allowing a user to select a subject in a display image displayed on a display unit based on the visible light image, 2. The imaging apparatus according to claim 1, wherein the recording determination means determines whether or not to record temperature information of the subject based on the detection result, the identification result, and a selection made by the user through the operation means.

5. The imaging device according to claim 4, further comprising a second display control means for superimposing, on the display image, second display information indicating that the subject is one that the user can select using the operation means, based on the detection result and the identification result.

6. an operation means for a user to perform a priority type setting for setting a type of subject for which temperature information is to be recorded with priority; 2. The image pickup apparatus according to claim 1, wherein the recording determination means determines whether or not to record temperature information of the subject based on the priority type setting in addition to the detection result and the identification result.

7. an operation means for a user to carry out authentication setting for registering in advance a subject whose temperature information is to be recorded; The imaging device according to claim 1, characterized in that the identification means obtains, as the identification result, an authentication result indicating whether one or more subjects detected by the detection means are identical to subjects pre-registered in the authentication setting.

8. 8. The imaging device according to claim 4, wherein the operation means is capable of performing a touch operation, a button operation, a key operation, or a stick operation.

9. 2. The imaging apparatus according to claim 1, further comprising a detection and identification means which serves as both the detection means and the identification means.

10. A first imaging means for acquiring a visible light image; A second imaging means for acquiring a non-visible light image; a detection means for detecting one or more objects based on the visible light image to obtain one or more detection results; an operation means for allowing a user to select a subject in a display image displayed on a display unit based on the visible light image; a record determination means for determining whether or not to record temperature information of a subject based on a selection made by a user using the operation means; and an acquisition and recording means for, when the recording determination means determines that the temperature information of the subject should be recorded, acquiring the temperature information of the subject that is the subject of the determination by the recording determination means based on the non-visible light image and the detection result, and recording the information on a recording medium.

11. 11. The imaging device according to claim 10, wherein the operation means allows touch operation, button operation, key operation, or stick operation.

12. The imaging device according to claim 10, further comprising a first display control means for superimposing, on the display image, first display information indicating that the subject is one about which temperature information is recorded on the recording medium by the acquisition and recording means, based on the detection result and a user selection made using the operation means.

13. The imaging device according to claim 1 or 10, characterized in that the acquisition and recording means links central position information and area information of the subject included in the detection result with temperature information acquired from the non-visible light image based on the central position information and the area information and records them on the recording medium.

14. The imaging apparatus according to claim 13 , wherein the detection means obtains the size information, the central position information, and the area information of the subject as the detection result by using machine learning.

15. The imaging device according to claim 1 or 10, characterized in that the acquisition and recording means acquires and averages multiple pieces of temperature information of the subject that is the subject of judgment by the recording and judgment means, and records the averaged temperature information of the subject on the recording medium.

16. 16. The imaging device according to claim 15, wherein the acquisition and recording means acquires a plurality of pieces of temperature information of the subject that is the subject of judgment by the record judgment means in accordance with a shutter speed or an exposure time of the imaging device.

17. 16. The imaging device according to claim 15, wherein the acquisition and recording means acquires a plurality of pieces of temperature information of the subject that is the subject of judgment by the record judgment means, the temperature information being acquired over a period before and after a shutter release time or before and after an exposure time of the imaging device.

18. a tracking means for tracking a subject that has been determined by the recording determination means, The imaging device according to claim 1 or 10, characterized in that the acquisition and recording means continuously acquires and averages temperature information of the subject being tracked by the tracking means while the tracking means is tracking, and records the averaged temperature information of the subject on the recording medium.

19. 19. The imaging device according to claim 18, wherein, when the tracking means loses sight of the subject being tracked while the tracking means is tracking the subject, the acquisition and recording means erases temperature information of the subject being tracked that was continuously acquired until the tracking means lost sight of the subject being tracked.

20. a first imaging step of acquiring a visible light image; A second imaging step of acquiring a non-visible light image; a detection step of detecting one or more objects based on the visible light image to obtain one or more detection results; a classification step of acquiring one or more classification results by classifying the one or more types of subjects detected in the detection step based on the visible light image; a recording determination step of determining whether or not to record temperature information of the subject based on the detection result and the identification result; and an acquisition and recording step of, when the recording determination step determines that the temperature information of the subject should be recorded, acquiring temperature information of the subject determined in the recording determination step based on the non-visible light image and the detection result, and recording the temperature information on a recording medium.

21. A program for causing a computer to execute each of the means of the imaging apparatus according to claim 1.

Images