Display control device, display control method, imaging system, and program

Abstract

Provided are a display control device, a display control method, an imaging system, and a program which make it possible to intelligibly present a region of a main subject in a visible light image by using an infrared image. This display control device comprises a processor that acquires a visible light image obtained through imaging in a wavelength region including the visible light range, acquires an infrared image obtained through imaging in a wavelength region including the infrared range, displays the visible light image, detects a region of a main subject on the basis of temperature information of a subject included in the infrared image, and changes gradation characteristics in display of a region of the visible light image corresponding to the region.

Description

Display control device, display control method, photographing system, and program

[0001] The present disclosure relates to a display control device, a display control method, an imaging system, and a program, and more particularly to a technique for reflecting information obtained from an infrared image in the display of a visible light image.

[0002] Patent Document 1 describes a subject image extraction method comprising: means for capturing an image of a subject and outputting a temperature distribution image; means for capturing the image of the subject and outputting a visible image; means for matching the position of the subject image in the output images of both means; binarization means for converting the temperature distribution image from the temperature distribution image output means into a binary signal using a predetermined temperature as a threshold; and image gate means for using the binary signal as a sampling signal to extract only the subject image from the output visible image.

[0003] Patent document 2 describes an image synthesis device that includes a first acquisition means for acquiring a background image and first distance information indicating the distance between a main subject included in the background image and an imaging device that captured the background image; a second acquisition means for acquiring a foreground image and second distance information indicating the distance between a main subject included in the foreground image and an imaging device that captured the foreground image; a comparison means for comparing the first distance information with the second distance information; a processing means for applying a predetermined image processing to at least one of the background image and the foreground image based on the comparison result by the comparison means; and a synthesis means for synthesizing the background image and the foreground image, at least one of which has been subjected to the predetermined image processing by the processing means.

[0004] JP-A-2-58480 Patent No. 7188397

[0005] One embodiment of the technique of the present disclosure provides a display control device, a display control method, an imaging system, and a program that can clearly present the area of ​​a main subject in a visible light image using an infrared image.

[0006] A display control device according to a first aspect of the present disclosure includes a processor that acquires a visible light image obtained by imaging in a wavelength range including the visible light range, acquires an infrared image obtained by imaging in a wavelength range including the infrared range, displays the visible light image, detects the area of ​​a main subject based on temperature information of the subject contained in the infrared image, and changes the gradation characteristics in the display of the area of ​​the visible light image corresponding to the area.

[0007] A display control device according to a second aspect may be configured in the first aspect such that the processor changes the gradation characteristics of at least an area other than the main subject in the visible light image based on the temperature information and displays the visible light image.

[0008] The display control device according to the third aspect may be configured such that in the first or second aspect, the processor differentiates the gradation characteristics between a first region, which is a region of a main subject in a visible light image, and a second region, which is a region other than the main subject.

[0009] A display control device according to a fourth aspect may be configured as in the third aspect, wherein the processor changes the gradation characteristics of the second region in accordance with the brightness distribution of the main subject in the visible light image.

[0010] A display control device according to a fifth aspect may be configured in the third or fourth aspect such that the processor changes the gradation characteristics of the second region based on an index value relating to the brightness of the region of the main subject in the visible light image.

[0011] A display control device according to a sixth aspect may be configured in any one of the third to fifth aspects, wherein the processor accepts a designation of a gradation setting to be applied to the display of the second area, and changes the gradation characteristics of the second area in accordance with the designated gradation setting.

[0012] A display control device according to a seventh aspect may be configured in any one of the first to sixth aspects, wherein the processor detects the area of ​​the main subject according to a preset temperature condition.

[0013] The display control device according to the eighth aspect may be configured in any one of the first to seventh aspects, wherein the processor detects an area in the infrared image that satisfies specific temperature conditions as the area of ​​the main subject based on temperature information.

[0014] A display control device according to a ninth aspect may be configured as in the eighth aspect, wherein the processor detects an area in the infrared image showing a temperature between 30°C and 42°C as the area of ​​the main subject based on the temperature information.

[0015] A display control device according to a tenth aspect may be configured in any one of the first to ninth aspects, wherein the processor accepts specification of a set temperature range to be applied to detection of the area of ​​the main subject, and detects the area in the infrared image within the specified set temperature range as the area of ​​the main subject.

[0016] A display control device according to an eleventh aspect may be configured in any one of the first to tenth aspects, wherein the processor changes the gradation characteristics by changing the gamma value of the gradation characteristics in the visible light image.

[0017] A display control device according to the twelfth aspect may be configured in the eleventh aspect such that one of the gradation characteristics applied to the area of ​​the main subject in the visible light image and the gradation characteristics applied to the area other than the main subject has a gamma value of 1 or more, and the other has a gamma value of less than 1.

[0018] A photography system according to a thirteenth aspect of the present disclosure includes a first image acquisition unit that acquires a visible light image obtained by imaging in a wavelength range including the visible light range, a second image acquisition unit that acquires an infrared image obtained by imaging in a wavelength range including the infrared range, a display unit that displays the visible light image, and a processor that detects the area of ​​the main subject based on temperature information of the subject included in the infrared image and changes the gradation characteristics in the display of the area of ​​the visible light image that corresponds to the area.

[0019] The imaging system according to a fourteenth aspect may be configured as in the thirteenth aspect, wherein the first image acquisition unit includes a visible light camera, and the second image acquisition unit includes an infrared camera.

[0020] The photographing system according to a fifteenth aspect may be configured as in the fourteenth aspect, wherein the visible light camera includes a display unit and a processor.

[0021] The imaging system according to a sixteenth aspect may be configured as in the fourteenth or fifteenth aspect, wherein the infrared camera is used by being connected to a visible light camera.

[0022] A display control method according to a seventeenth aspect of the present disclosure is a display control method executed by a processor, and includes the processor acquiring a visible light image obtained by imaging in a wavelength range including the visible light range and an infrared image obtained by imaging in a wavelength range including the infrared range, displaying the visible light image, detecting the area of ​​a main subject based on temperature information of the subject contained in the infrared image, and performing processing to change the gradation characteristics in displaying the area of ​​the visible light image corresponding to the area.

[0023] A program according to an eighteenth aspect of the present disclosure enables a computer to perform the following functions: acquire a visible light image obtained by imaging in a wavelength range including the visible light range and an infrared image obtained by imaging in a wavelength range including the infrared range; display the visible light image; detect the area of ​​a main subject based on temperature information of the subject contained in the infrared image; and change the gradation characteristics in displaying the area of ​​the visible light image corresponding to the area.

[0024] The present disclosure also includes a tangible, non-transitory, computer-readable recording medium such as a CD-ROM (Compact Disk-Read Only Memory) that stores the program according to the eighteenth aspect.

[0025] FIG. 1 is a block diagram schematically illustrating a configuration of an imaging system including a display control device according to an embodiment of the present disclosure. FIG. 2 is a flowchart illustrating an example of a display control method executed by a processor. FIG. 3 is an explanatory diagram illustrating an example of a display image generated by changing the gradation characteristics of a visible light image based on the temperature distribution of an infrared image. FIG. 4 is an explanatory diagram illustrating an example of a setting screen for accepting designation of a set temperature range. FIG. 5 is a graph illustrating an example of input / output characteristics of gamma correction. FIG. 6 is an example of a display image generated by changing the gradation characteristics of an area other than a main subject based on the brightness of the main subject included in the visible light image. FIG. 7 is another example of a display image generated by changing the gradation characteristics of an area other than a main subject based on the brightness of the main subject included in the visible light image. FIG. 8 is an explanatory diagram illustrating an example of an image density setting screen for setting the display density of an area other than a main subject. FIG. 9 is a block diagram illustrating the functional configuration of a display control device. FIG. 10 is a perspective view illustrating a specific example of an imaging system. FIG. 11 is a rear view of the imaging system illustrated in FIG. 10. FIG. 12 is an explanatory diagram illustrating an example of the relationship between the angle of view of a visible light camera and the angle of view of an infrared camera. FIG. 13 is a block diagram illustrating an example of the internal configuration of an imaging system. FIG. 14 is a flowchart showing an example of a density change process for a visible light image in a live view display.

[0026] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] 1 is a block diagram showing a schematic configuration of an imaging system 10 including a display control device 2 according to an embodiment of the present disclosure. The display control device 2 includes a processor 4 and a storage device 6. The imaging system 10 includes a visible light image acquisition unit 12, an infrared image acquisition unit 14, the display control device 2, a display unit 16, and an operation unit 18.

[0028] The processor 4 includes a CPU (Central Processing Unit). The processor 4 may include a GPU (Graphics Processing Unit). The processor 4 may be configured to include a digital signal processor and / or a programmable logic device. The storage device 6 is a tangible, non-transitory computer-readable medium, and may be, for example, a semiconductor memory, a hard disk drive, a solid-state drive, or a combination of these. The storage device 6 may be configured to include, for example, a memory serving as a main storage device and a storage serving as an auxiliary storage device. The processor 4 functions as a processing unit that performs various processes by executing instructions stored in the memory.

[0029] The visible light image acquisition unit 12 acquires a visible light image VSI obtained by capturing an image in a wavelength range including the visible light range. The visible light range refers to the wavelength range of electromagnetic waves from 360 nm to 800 nm. The visible light image acquisition unit 12 may be configured to include a visible light camera that captures visible light. A visible light camera, which is a visible light imaging device, typically includes an imaging optical system including one or more lenses and a visible light imaging element (hereinafter referred to as a visible light sensor) that captures an optical image formed by the imaging optical system and converts it into an electrical signal. The visible light camera may be configured to include an image processing circuit that processes the electrical signal obtained from the visible light sensor to create a digital image. The visible light sensor may be configured to detect light in the visible light range and may have sensitivity in part of the near-infrared wavelength range. The visible light image acquisition unit 12 is an example of a "first image acquisition unit" in this disclosure.

[0030] The infrared image acquisition unit 14 acquires an infrared image IRI by capturing an image in a wavelength range including the infrared range. The infrared range refers to the wavelength range of electromagnetic waves from 0.8 μm to 1000 μm. The infrared image acquisition unit 14 may be configured to include an infrared camera that captures infrared light. An infrared camera, which is an infrared imaging device, typically includes an imaging optical system including one or more lenses and an infrared imaging element (hereinafter referred to as an infrared sensor) that captures an optical image formed by the imaging optical system and converts it into an electrical signal. The infrared camera may also include an image processing circuit that processes the electrical signal obtained from the infrared sensor to generate a digital image. The infrared camera encompasses the concepts of a thermography device and a thermal camera that capture a heat map image that visualizes the temperature distribution of a subject. A heat map image may also be referred to as a thermography image (thermal image). The infrared image IRI may be an image before heat mapping or a heat map image. In this specification, the term "infrared image IRI" includes the concept of a heat map image, unless the context indicates otherwise.

[0031] The infrared sensor may be a sensor having high sensitivity to any wavelength region among the near-infrared region (0.8 μm to 2.5 μm), the mid-infrared region (2.5 μm to 4.0 μm), and the far-infrared region (4.0 μm to 1000 μm). Note that, as an infrared sensor corresponding to the far-infrared region, for example, a sensor having sensitivity in the range of 4.0 μm to 14 μm can be used. The infrared sensor can be a thermal infrared sensor such as a microbolometer or an SOI (Silicon on Insulator) diode type. The infrared image acquisition unit 14 is an example of a "second image acquisition unit" in the present disclosure.

[0032] The resolution of the infrared sensor and the resolution of the visible light sensor may be the same or different. For example, the resolution of the infrared sensor may be lower than the resolution of the visible light sensor. The angle of view of the infrared camera and the angle of view of the visible light camera may be the same or different. For example, the angle of view of the infrared camera may be wider than the angle of view of the visible light camera.

[0033] In the imaging system 10 of this embodiment, the infrared image IRI and the visible light image VSI are acquired by capturing images such that a common subject is included in each of the infrared image IRI and the visible light image VSI. Preferably, the entire imaging range of the visible light image VSI is included in the imaging range of the infrared image IRI, but a configuration may also be adopted in which a portion of the imaging range of the visible light image VSI is included in the imaging range of the infrared image IRI so that at least a portion of the subject included in the visible light image VSI and the subject included in the infrared image IRI overlap.

[0034] The correspondence between the pixel positions on the infrared image IRI and the pixel positions on the visible light image VSI can be determined based on, for example, the configurations of the imaging optical systems of the visible light camera and the infrared camera, the spatial arrangement between them, etc. Furthermore, the correspondence between pixel positions between these two images can be determined based on information about corresponding points of the subject that are common to both images.

[0035] The photographing system 10 is not limited to a system configuration that combines a visible light camera and an infrared camera, but may also be configured to be capable of acquiring a visible light image VSI and an infrared image IRI of the same imaging range at the same angle of view by splitting the optical path of the imaging optical system, such as a single-lens camera equipped with both a visible light sensor and an infrared sensor.

[0036] It is preferable that the visible light image VSI and the infrared image IRI are captured approximately simultaneously, but they are not limited to being captured strictly simultaneously, and may be captured with a time difference within an allowable range for positional deviation of a common subject between the images due to a time difference in the capture timing of the two images. For example, the visible light image VSI and the infrared image IRI may be captured within a time period that can be considered to be approximately simultaneous, including a time difference within a range that can be perceived as substantially simultaneous by human perception.

[0037] The visible light image VSI and the infrared image IRI acquired via the visible light image acquisition unit 12 and the infrared image acquisition unit 14 may be either still images or moving images.

[0038] The display unit 16 may be, for example, a liquid crystal display, an organic electroluminescence (OEL) display, a projector, or an appropriate combination of these display devices. The display unit 16 may be, for example, a monitor disposed on the back of the visible light camera body, an electronic viewfinder, or a combination of these. The display unit 16 may also be a monitor of a personal computer, or a touch panel display of a mobile terminal such as a smartphone or a tablet terminal.

[0039] The operation unit 18 is a user interface for the user to input various instructions and information, and is configured by various input devices such as operation buttons, dials, switches, levers, joysticks, touch panels, keyboards, mice, other pointing devices, or voice input devices, or an appropriate combination of these. Note that the display unit 16 and the operation unit 18 may be configured integrally, such as a touch panel display.

[0040] [Outline of Display Control Method] FIG. 2 is a flowchart showing an example of a display control method executed by the processor 4.

[0041] In step ST1, the processor 4 acquires a visible light image VSI and an infrared image IRI.

[0042] In step ST2, the processor 4 acquires temperature information of the subject from the infrared image IRI. For example, the processor 4 may convert the infrared image IRI into temperature information to generate a heat map image representing the temperature distribution of the subject. Note that the processor 4 may acquire the heat map image as the infrared image IRI.

[0043] In step ST3, the processor 4 detects an area of ​​pixels in the infrared image IRI that falls within a set temperature range as the area of ​​the main subject based on the temperature information of the subject contained in the infrared image IRI. The set temperature range is set as the temperature condition of the subject to be detected, and corresponds to the type of subject to be detected. For example, if the subject to be detected is a human or an animal such as a dog or cat, the set temperature range may be set to 35°C to 38°C. Alternatively, if the subject to be detected is a bird, the set temperature range may be set to 39°C or higher.

[0044] The set temperature range may be a specific temperature range set in advance, such as 30°C to 42°C, or may be configured so that the corresponding set temperature range is set by selecting the type of subject based on a table in which the type of subject to be detected is linked to the set temperature range, or may be configured so that the set temperature range is set by the user freely specifying the numerical value of the temperature range via a user interface. A temperature condition determined by a set temperature range is an example of a "specific temperature condition" in the present disclosure.

[0045] Processor 4 detects, as the area of ​​the main subject, an area (image area) of pixels on infrared image IRI that satisfies the temperature condition of a set temperature range. Furthermore, processor 4 identifies an area on visible light image VSI that corresponds to the area of ​​the main subject detected from infrared image IRI, based on the correspondence between pixel positions on infrared image IRI and pixel positions on visible light image VSI. That is, processor 4 can detect the corresponding area of ​​the main subject from visible light image VSI based on temperature information of the subject obtained from infrared image IRI. The area of ​​the main subject may be detected pixel by pixel, or may be detected as a shape such as a circumscribing rectangle surrounding the main subject.

[0046] In step ST4, the processor 4 changes the gradation characteristics for displaying the visible light image VSI that includes an area corresponding to the area of ​​the main subject detected from the infrared image IRI. For example, the processor 4 applies different gradation characteristics to the area of ​​the main subject in the visible light image VSI that corresponds to the area of ​​the main subject detected from the infrared image IRI and to areas other than the main subject in the visible light image VSI.

[0047] The gradation characteristics can be expressed as a density conversion function applied to a density conversion process (density gradation conversion process) of pixel values. The density conversion function may be a table that defines the correspondence between input pixel values ​​and output pixel values. This type of density conversion process is called gamma correction, and the input / output characteristics can be changed by changing the gamma value. Changing the gamma value is equivalent to changing the gradation characteristics.

[0048] Preferably, processor 4 changes the gradation characteristics applied to at least the areas other than the main subject in the visible light image VSI, thereby varying the display density between the area of ​​the main subject and the areas other than the main subject, thereby relatively increasing the visibility of the main subject in the visible light image VSI. Display density may also be referred to as "brightness" in the display. "Brightness" or "density" may be expressed by the values ​​(pixel values) of the pixels constituting the visible light image VSI. For example, if the visible light image VSI is an 8-bit digital image of each of R (red), G (green), and B (blue), the pixel values ​​of each RGB color may range from 0 to 255. A pixel value of 0 represents the darkest value, and a pixel value of 255 represents the brightest value. Preferably, processor 4 changes the display density of the areas other than the main subject in accordance with the brightness distribution of the main subject included in the visible light image VSI. Note that processor 4 may also change the display density of the areas other than the main subject according to a gradation setting specified by the user.

[0049] In step ST5, processor 4 applies the gradation characteristics changed in step ST4 to perform density conversion of pixel values ​​of visible light image VSI to generate an image for display. Processor 4 performs density conversion on pixel values ​​of at least areas other than the main subject in visible light image VSI so that they are brighter or darker than the area of ​​the main subject, generating an image for display having a difference in display density (difference in brightness) between the area of ​​the main subject and the areas other than the main subject.

[0050] In step ST6, the processor 4 displays the display image generated in step ST5 on the display unit 16. After step ST6, the processor 4 ends the flowchart in Fig. 2. Note that the processes of steps ST1 to ST6 may be repeatedly executed for the visible light images VSI and the infrared images IRI acquired in time series.

[0051] For example, when performing live view display in which moving images being captured are displayed on the display unit 16, the flowchart of FIG. 2 may be executed for each frame of the moving image.

[0052] [Example of Display Image] Figure 3 is an explanatory diagram showing an example of a display image DSI1 generated by changing the gradation characteristics of the visible light image VSI1 based on the temperature distribution of the infrared image IRI1. The visible light image VSI1 shown in the upper left of Figure 3 is an example of a visible light image captured using a visible light camera. The infrared image IRI1 shown in the lower left of Figure 3 is an example of an infrared image obtained by capturing the same subject as the visible light image VSI1 using an infrared camera.

[0053] The area of ​​main subject MS1 is detected based on a set temperature range from the temperature distribution of the subject included in infrared image IRI1, and the corresponding area of ​​main subject MSv1 is detected in visible light image VSI1 based on position information of the area of ​​main subject MS1. Here, an example is shown in which the area of ​​a cat, which is main subject MS1, is detected from infrared image IRI1. This makes it possible to divide visible light image VSI1 into an area of ​​main subject MSv1 and an area other than main subject MSv1. The area of ​​main subject MSv1 is an example of a "first area" in the present disclosure, and the area other than main subject MSv1 is an example of a "second area" in the present disclosure.

[0054] FIG. 3 shows an example of a display image DSI1 in which the gamma value applied to the density conversion of pixel values ​​in areas other than the main subject MSv1 in the visible light image VSI1 has been changed, and the display density of areas other than the main subject MSv1 has become brighter than before the density conversion.

[0055] [Example of a Setting Screen for Accepting Designation of a Set Temperature Range] Fig. 4 is an example of a setting screen for accepting designation of a set temperature range. A setting screen such as that shown in Fig. 4 is displayed on the display unit 16 under the control of the processor 4. The display control device 2 is preferably configured to allow the user to arbitrarily change the set temperature range, which is the temperature threshold that defines the temperature conditions of the detection target. For example, as shown in Fig. 4, it is preferable that a reference set temperature be displayed for each type of subject to be detected.

[0056] For example, if humans and / or animals are to be detected, the temperature range of "humans and animals (M: 35-38 degrees)" is selected. If birds are to be detected, the temperature range of "birds (H: 39 degrees or higher)" is selected. When "user specified" is selected, a numeric input box is displayed, allowing the user to specify any temperature range. The temperature range specified by the user is stored in the storage device 6 and may be added to the menu as a set temperature range specified by the user.

[0057] It is also preferable that multiple set temperature ranges be selectable. For example, multiple types of subjects can be selected as detection targets, and these can be set in combination with user specifications. When multiple temperature ranges are specified, the set temperature range is the union of all the specified temperature ranges. For example, on the setting screen shown in FIG. 4, if both "Human / Animal (M: 35-38 degrees)" and "User Specified" are selected and "30-37 degrees" is specified by the user, the set temperature range is set to the union of these, i.e., a temperature range of 30°C to 38°C.

[0058] Alternatively, if both "Bird (H: 39 degrees or higher)" and "User Specified" are selected on the setting screen shown in FIG. 4 and the user specifies, for example, "20 to 25 degrees," the set temperature range is set to the union of these, that is, 20 to 25 degrees Celsius and 39 degrees Celsius or higher.

[0059] Alternatively, if "Humans and Animals (M: 35-38 degrees)," "Birds (H: 39 degrees or higher)," and "User Specified" are all selected on the setting screen shown in FIG. 4, and the user specifies, for example, "33-36 degrees," the set temperature range is set to the union of these values, a temperature range of 33°C or higher.

[0060] The upper temperature threshold for detecting birds does not need to be set, or a specific value such as 43°C may be set.

[0061] The setting screen may also display a "Detection OFF" option, which disables detection of the main subject using the infrared image IRI. Setting Detection OFF disables the process of controlling the display density of the visible light image based on the temperature distribution of the infrared image IRI. This allows the visible light image VSI to be displayed as it was before the density change.

[0062] [Regarding Image Density Conversion (Gamma Correction)] Fig. 5 is a graph showing an example of input / output characteristics of gamma correction. The horizontal axis represents the input tone value, and the vertical axis represents the output tone value. Gamma correction for an image with 256 tones expressed as values ​​from 0 to 255 converts the tone value X before correction into the tone value Y after correction using the following equation (1): Y = 255 x (X / 255) 1 / γ (1)

[0063] When γ>1, the image becomes bright overall, and when γ<1, the image becomes dark overall.

[0064] By applying gamma correction to each of the RGB signal values ​​of the visible light image VSI, pixel values ​​are converted and an image for display is generated. Note that pixel values ​​can be understood as pixel density values. In this embodiment, gamma correction is performed by applying different gamma values ​​to the area of ​​the main subject and the areas other than the main subject in the visible light image VSI, thereby generating an image for display by providing a density difference between the area of ​​the main subject and the areas other than the main subject. By displaying the image for display generated in this manner on the display unit 16, the visibility of the area of ​​the main subject can be relatively improved.

[0065] In the default setting of the display control device 2 according to the embodiment, pixel values ​​in the area of ​​the main subject in the visible light image VSI are left unchanged (no density conversion), and the gamma value is changed to perform gamma correction (conversion) on pixel values ​​in areas other than the main subject. For example, in the default setting, if the average density of the main subject is greater than a threshold, the gamma value applied to convert the density values ​​(pixel values) of areas other than the main subject is set to γ ​​= a < 1, and if the average density of the main subject is less than the threshold, the gamma value is set to γ ​​= b > 1. a may be set to, for example, 0.4, and b may be set to, for example, 2.5. By changing the gamma value in this way, the density difference between the area of ​​the main subject and areas other than the main subject can be increased.

[0066] Furthermore, rather than being limited to the default setting, it is preferable that the display density of the areas other than the main subject be freely set by the user so that the user can set a density that is easy to see. In the case where the display density of the areas other than the main subject is specified by the user, the user may freely set the density value within the range of 0 to 255, or may specify a gamma value to be applied to the conversion of pixel values ​​of the areas other than the main subject. For example, the processor 4 accepts, via the operation unit 18, a specification of a gradation setting to be applied to the display of the areas other than the main subject, and changes the gradation characteristics of the areas other than the main subject according to the specified gradation setting.

[0067] 6 and 7 show examples of display images in which the display density (gradation characteristics) of areas other than the main subject is changed according to the brightness distribution of the main subject included in the visible light image. Fig. 6 shows an example of a display image DSI2 that is generated when the brightness of the main subject MSv2 included in the visible light image VSI2 is brighter than a threshold. Because the area of ​​the white cat that is the main subject MSv2 in the visible light image VSI2 has large pixel values ​​and is bright, it is preferable to change the gradation characteristics and perform density conversion so that the area other than the main subject MSv2 is displayed darker.

[0068] For example, in the default setting, if the average density of the main subject MSv2 is greater than a threshold, the gamma value applied to density conversion of pixel values ​​in areas other than the main subject MSv2 is changed to a value smaller than 1, such as γ=0.4. The average density of the main subject MSv2 can be calculated as the average value of pixel values ​​in the area of ​​the main subject MSv2. The threshold value compared with the average density may be, for example, 127.5, which is the central value of 256 gradations. Note that the average density is an example of an index value related to the brightness of the main subject MSv2. Instead of the average density, for example, other representative values ​​such as the median or mode, or a new index value defined by combining multiple statistics, may be used.

[0069] In this way, the display image DSI2 generated by performing density conversion processing with the changed gamma value (γ=0.4) displays areas other than the main subject MSv2 darkly, improving the visibility of the main subject MSv2.

[0070] 7 shows an example of a display image DSI3 that is generated when the brightness of the main subject MSv3 included in the visible light image VSI3 is darker than a threshold value. Because the area of ​​the black cat that is the main subject MSv3 in the visible light image VSI3 has small pixel values ​​and is dark, it is preferable to change the gradation characteristics and perform density conversion so that the area other than the main subject MSv3 is displayed brighter.

[0071] For example, in the default setting, if the average density of the main subject MSv3 is smaller than the threshold value, the gamma value applied to density conversion of pixel values ​​in areas other than the main subject MSv3 is changed to a value greater than 1, for example, γ=2.5.

[0072] In this way, the display image DSI3 generated by performing density conversion processing with the changed gamma value (γ=2.5) displays areas other than the main subject MSv3 brightly, improving the visibility of the main subject MSv3.

[0073] [Example of a User Interface Screen for Setting the Display Density of Areas Other Than the Main Subject] Figure 8 is an explanatory diagram showing an example of an image density setting screen 30 for setting the display density of areas other than the main subject. Diagram F8A on the left side of Figure 8 shows an example of the screen when the display density of areas other than the main subject MSv4 is set to level 1, the darkest level of the seven levels, using brightness adjustment bar 32, which can be changed in seven levels from level 1 to level 7. Diagram F8B in the center of Figure 8 shows an example of the screen when set to level 4, and diagram F8C on the right side shows an example of the screen when set to level 7.

[0074] As shown in FIG. 8 , the image density setting screen 30 displays a brightness adjustment bar 32 below the image display area 34. An inverted triangle mark 36 displayed above the brightness adjustment bar 32 indicates the currently selected level. Levels 1 to 7 may correspond to, for example, the seven gamma values ​​shown in FIG. 4 . The image display area 34 displays an image when the brightness is set to the currently selected level. The user can check the image displayed in the image display area 34 and select a desired brightness level via the operation unit 18. After selecting the desired brightness level, the user can press the set button 38 to set the display density of areas other than the main subject MSv4. Note that the method of setting the density in stages is not limited to this, and a user interface that allows for free specification of a density conversion curve (gamma curve) may also be employed.

[0075] Although not shown in the drawings, the display density of the area of ​​the main subject MSv4 may also be configured to be settable in a similar manner. Note that the setting of the display density of the area of ​​the main subject MSv4 may be the same as the setting of the display density when the visible light image VSI is displayed directly on the display unit 16 without using information from the infrared image IRI.

[0076] 9 is a block diagram showing the functional configuration of the display control device 2. The processor 4 functions as an infrared image receiving unit 40, a temperature information conversion unit 42, a temperature range setting unit 44, a main subject region detection unit 46, a visible light image receiving unit 50, a corresponding region detection unit 51, a density evaluation unit 52, a gradation characteristic control unit 54, a density conversion unit 56, and a display image output unit 58.

[0077] The infrared image receiving unit 40 receives input of an infrared image IRI captured using an infrared sensor. The temperature information conversion unit 42 converts the infrared image IRI acquired via the infrared image receiving unit 40 into temperature information. The temperature information conversion unit 42 may be configured to generate, for example, a heat map image showing the temperature distribution of the subject included in the infrared image IRI. The heat map image generated based on the infrared image IRI is an image obtained by converting the infrared image IRI into temperature information, and can essentially be understood as an "infrared image."

[0078] The temperature range setting unit 44 sets a temperature range for a target to be detected as a main subject among the subjects included in the infrared image IRI. The temperature range setting unit 44 sets, for example, the union of the temperature conditions specified on the setting screen for the set temperature range described with reference to FIG. 4 as the set temperature range.

[0079] The main subject region detection unit 46 detects, as the main subject region, a region of the subject that falls within the set temperature range from the infrared image IRI, based on the temperature distribution of the subject included in the infrared image IRI and the set temperature range set by the temperature range setting unit 44. Information on the main subject region detected by the main subject region detection unit 46 is provided to the corresponding region detection unit 51 and the gradation characteristic control unit 54.

[0080] The visible light image receiving unit 50 receives input of the visible light image VSI captured using a visible light sensor. The corresponding area detection unit 51 identifies an area in the visible light image VSI that corresponds to the area of ​​the main subject detected by the main subject area detection unit 46. That is, the corresponding area detection unit 51 identifies the corresponding area of ​​the main subject in the visible light image VSI based on information about the area of ​​the main subject detected by the main subject area detection unit 46 and information about the correspondence between the two images. The combination of the main subject area detection unit 46 and the corresponding area detection unit 51 detects the area of ​​the main subject from the visible light image VSI based on temperature information from the infrared image IRI.

[0081] The density evaluation unit 52 evaluates the density of the area of ​​the main subject in the visible light image VSI detected by the corresponding area detection unit 51. The density evaluation unit 52 calculates an index value related to brightness, such as the average value of pixel values ​​(average density) of the area of ​​the main subject, and compares it with a threshold value.

[0082] The gradation characteristics control unit 54 controls the gradation characteristics to be applied to the display of the visible light image VSI. The gradation characteristics control unit 54 includes a main subject region gradation characteristics modification unit 60 and a background region gradation characteristics modification unit 62. The main subject region gradation characteristics modification unit 60 performs processing to modify the main subject region gradation characteristics TC1 to be applied to the display of the region of the main subject in the visible light image VSI. The main subject region gradation characteristics TC1 may be gradation characteristics to be applied to the display of the entire image when the visible light image VSI is displayed on the display unit 16 when processing to control the display density of the visible light image VSI based on detection of the region of the main subject using the infrared image IRI is not performed.

[0083] The background region gradation characteristics modification unit 62 performs processing to modify the background region gradation characteristics TC2 applied to display of regions other than the main subject in the visible light image VSI. Note that the term "background region" refers to regions other than the main subject. The background region gradation characteristics modification unit 62 modifies the background region gradation characteristics TC2 based on the evaluation results of the density evaluation unit 52. The background region gradation characteristics modification unit 62 can also modify the background region gradation characteristics TC2 in accordance with a density setting specified by the user.

[0084] The density conversion unit 56 includes a main subject region density conversion unit 64 and a background region density conversion unit 66. The main subject region density conversion unit 64 applies the main subject region gradation characteristics TC1 to perform density conversion on the region of the main subject in the visible light image VSI. The background region density conversion unit 66 applies the background region gradation characteristics TC2 to perform density conversion on the region other than the main subject in the visible light image VSI. After processing by the density conversion unit 56, the display image DSI is generated.

[0085] The display image output unit 58 performs output processing of the display image DSI. The display image output unit 58 converts the display image DSI into a signal format suitable for display on the display unit 16 and outputs it to the display unit 16. In this way, the display image DSI is displayed on the display unit 16. The display control device 2 may also be understood as an image processing device that processes the visible light image VSI and the infrared image IRI.

[0086] [Hardware Configuration of Each Processing Unit] The hardware configuration of processing units that execute various processes, such as the infrared image receiving unit 40, temperature information conversion unit 42, temperature range setting unit 44, main subject region detection unit 46, visible light image receiving unit 50, corresponding region detection unit 51, density evaluation unit 52, gradation characteristics control unit 54, density conversion unit 56, display image output unit 58, main subject region gradation characteristics change unit 60, background region gradation characteristics change unit 62, main subject region density conversion unit 64, and background region density conversion unit 66, which are described in FIG. 9, is made up of various processors as shown below.

[0087] Various types of processors include CPUs, which are general-purpose processors that execute programs and function as various processing units, GPUs, which are processors specialized for image processing, programmable logic devices (PLDs), such as FPGAs (Field Programmable Gate Arrays), which are processors whose circuit configuration can be changed after manufacture, and dedicated electrical circuits, such as ASICs (Application Specific Integrated Circuits), which are processors with a circuit configuration designed specifically for executing specific processes.

[0088] A single processing unit may be composed of one of these various processors, or may be composed of two or more processors of the same or different types. For example, a single processing unit may be composed of multiple FPGAs, a combination of a CPU and an FPGA, or a combination of a CPU and a GPU. Multiple processing units may also be composed of a single processor. Examples of multiple processing units composed of a single processor include, first, a configuration in which a single processor is composed of a combination of one or more CPUs and software, as typified by computers such as client and server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system-on-chip (SoC). In this way, the various processing units are composed of one or more of the above-mentioned various processors as a hardware structure.

[0089] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0090] [Specific Example of the Imaging System 10] A more specific example of the imaging system 10 according to the embodiment of the present disclosure will be described.

[0091] For example, when photographing animals using imaging equipment such as a digital camera, if the animal is in the forest or between branches, it can be difficult to find it with the human eye alone and capture it within the field of view. Therefore, a method is used in which an infrared camera is used in conjunction with a visible-light camera to make it easier to find animals hiding in nature and assist in photographing animals. However, since photographers often take photos while viewing visible-light images, it is desirable to utilize information obtained from infrared images in the display of visible-light images to provide users with useful information.

[0092] 10 and 11 show a specific example of the imaging system 10 according to an embodiment of the present disclosure. Fig. 10 is a perspective view of the imaging system 10 as seen obliquely from the front, and Fig. 11 is a rear view of the imaging system 10 shown in Fig. 10. As shown in Fig. 10, the imaging system 10 includes a visible light camera 100 and an infrared camera 200.

[0093] The visible light camera 100 is a mirrorless digital single-lens camera that is composed of an interchangeable lens 102 and a camera body 104 to which the interchangeable lens 102 is detachable.

[0094] A body mount (not shown) for mounting the interchangeable lens 102 is provided on the front of the camera body 104. A shutter release button 105, a shutter speed dial 106, an exposure compensation dial 107, a power lever 108, a hot shoe 109, etc. are provided on the top surface of the camera body 104.

[0095] As shown in FIG. 11, the rear surface of the camera body 104 is provided with an EVF (Electronic View Finder) 112, a MENU / OK key 114, a cross key 116, a playback button 117, an LCD (Liquid Crystal Display) 118, and the like.

[0096] The LCD 118 functions as a display that displays a live view image in the shooting mode, plays back and displays a captured image in the playback mode, and also displays various menu screens. Note that in the shooting mode, when you bring your eye close to the EVF 112, an eye sensor (not shown) automatically switches the display to the EVF 112, and when you move your eye away, the display switches to the LCD 118.

[0097] The MENU / OK key 114 is an operation key that functions both as a menu button for issuing a command to display a menu on the screen of the LCD 118 and as an OK button for issuing a command to confirm and execute a selection.

[0098] The cross key 116 is an operation unit for inputting instructions in four directions (up, down, left, and right), and functions as a button for selecting an item from a menu screen or for instructing the selection of various setting items from each menu. The up and down keys of the cross key 116 function as a zoom switch during shooting or a playback zoom switch in playback mode, and the left and right keys function as frame-by-frame (forward and backward) buttons in playback mode. The playback button 117 is a button for switching to a playback mode in which captured and recorded still images or videos are displayed on the LCD 118.

[0099] The hot shoe 109 is an attachment portion for attaching an external accessory to the visible light camera 100. In the photography system 10, an infrared camera 200 is attached to the hot shoe 109. The visible light camera 100 and the infrared camera 200 are connected by a communication cable 122 capable of sending and receiving data.

[0100] The infrared camera 200 is an external infrared sensor module made up of an infrared lens 202 and a camera body 204 to which the infrared lens 202 is attached. The infrared lens 202 is a lens that selectively transmits and focuses light in the infrared range. The infrared camera 200 is used by connecting it to the visible light camera 100. When the infrared camera 200 is attached to the visible light camera 100, it is preferable that the optical axis of the infrared lens 202 is parallel to the optical axis of the interchangeable lens 102 of the visible light camera 100.

[0101] The angle of view of the infrared camera 200 is preferably wider than the angle of view of the visible light camera 100. It is sufficient that the imaging ranges of the visible light camera 100 and the infrared camera 200 overlap at least partially.

[0102] The infrared camera 200 may include a light source such as an infrared LED (Light Emitting Diode) that projects infrared rays toward the imaging range.

[0103] [Relationship between the angle of view of the visible light camera 100 and the angle of view of the infrared camera 200] Figure 12 is an explanatory diagram showing an example of the relationship between the angle of view AV of the visible light camera 100 and the angle of view AI of the infrared camera 200. The left diagram F12A of Figure 12 shows a case where the optical axis of the interchangeable lens 102 of the visible light camera 100 and the optical axis of the infrared lens 202 of the infrared camera 200 are parallel, the focal length of the interchangeable lens 102 and the focal length of the infrared lens 202 are the same, and the size of the image sensor 110 of the visible light camera 100 and the size of the image sensor 210 of the infrared camera 200 are the same. The angle of view AV and the angle of view AI have the same width, and the position of the angle of view AV and the position of the angle of view AI are shifted up and down depending on the amount of vertical shift between the optical axis of the interchangeable lens 102 and the optical axis of the infrared lens 202.

[0104] It is sufficient that the angle of view AV of the visible light camera 100 and the angle of view AI of the infrared camera 200 overlap at least partially. The central diagram F12B of Fig. 12 shows a case where the angle of view of the infrared camera 200 is wider than that of the visible light camera 100, and the optical axes of the two cameras are shifted up and down and left and right. The right diagram F12C of Fig. 12 shows a case where the angle of view of the infrared camera 200 is narrower than that of the visible light camera 100, and the optical axes of the two cameras are shifted up and down and left and right.

[0105] The angle of view AI of the infrared camera 200 may be fixed, for example, the angle of view of a lens equivalent to a focal length of 28 mm in a 35 mm format. The angle of view AI of the visible light camera 100 can be changed by changing the focal length of the interchangeable lens 102. The interchangeable lens 102 may be a zoom lens.

[0106] [Internal Configuration of Image Capturing System 10] Fig. 13 is a block diagram showing an example of the internal configuration of the image capturing system 10. As shown in Fig. 13, the visible light camera 100 includes an image sensor 110, a processor 120, a memory 130, a display driver 140, an operation unit 150, an input / output interface 160, a sensor driver 170, and an AFE (Analog Front End) 180. The processor 120, the memory 130, and the operation unit 150 correspond to the processor 4, the storage device 6, and the operation unit 18 in Fig. 1, respectively. In other words, the visible light camera 100 is a camera equipped with a display control device 2.

[0107] The image sensor 110 serving as a visible light sensor is configured by a CMOS (Complementary Metal-Oxide Semiconductor) color image sensor. Note that the image sensor 110 is not limited to a CMOS type, and may also be a CCD (Charge Coupled Device) type image sensor.

[0108] The image sensor 110 has red (R), green (G), and blue (B) color filters arranged in a periodic color array on a plurality of pixels each made up of a photoelectric conversion element (photodiode) arranged two-dimensionally in the x direction (horizontal direction) and y direction (vertical direction), and a microlens arranged on each photodiode. The periodic color array may be, for example, a Bayer array or X-Trans (registered trademark), etc.

[0109] An optical image of a subject formed on the light-receiving surface of the image sensor 110 by the imaging optical system of the interchangeable lens 102 is converted into an electrical signal by the image sensor 110. Charges corresponding to the amount of incident light are accumulated in each pixel of the image sensor 110, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 110 as an image signal.

[0110] The processor 120 performs overall control of each unit of the visible light camera 100 and various processes in accordance with user operations using the operation unit 150. The processes performed by the processor 120 include a process of changing the gradation characteristics in the display of a visible light image using an infrared image.

[0111] The memory 130 stores instructions to be executed by the processor 120. The memory 130 includes a flash memory (not shown), a random access memory (RAM) (not shown), and a read only memory (ROM) (not shown). The memory 130 also includes a memory card that is detachable from the camera body 104.

[0112] The flash memory and ROM are non-volatile memories that store firmware, various programs including a display control program according to an embodiment of the present disclosure, and captured images (still images, video images), etc.

[0113] The RAM functions as a working area for processing by the processor 120, and also temporarily stores firmware, display control programs, etc. stored in the non-volatile memory. Note that the processor 120 may have a portion of the memory 130 (RAM) built in.

[0114] The display driver 140 converts the input digital image signals into a signal format for display and sequentially outputs them to the EVF 112 or the LCD 118. Note that some or all of the processing functions of the display driver 140 may be included in the processor 120. The EVF 112 and the LCD 118 correspond to the display unit 16 in FIG. 1 .

[0115] The operation unit 150 includes the shutter release button 105, shutter speed dial 106, exposure compensation dial 107, power lever 108, MENU / OK key 114, cross key 116, playback button 117, etc. shown in FIGS.

[0116] 10 and 11 and a connector portion (not shown) to which the communication cable 122 is connected. The input / output interface 160 transmits and receives data and signals to and from the infrared camera 200.

[0117] The sensor driver 170 controls the reading of image signals from the image sensor 110 in accordance with instructions from the processor 120. The sensor driver 170 also has an electronic shutter function that, in response to an electronic shutter control signal from the processor 120, discharges (resets) the electric charges accumulated in each pixel of the image sensor 110 and starts exposure.

[0118] The AFE 180 performs various analog signal processing on an analog image signal obtained by capturing an image of a subject with the image sensor 110, and converts the processed image signal into a digital image signal. The analog processing in the AFE 180 includes, for example, color separation processing and AGC (Automatic Gain Control). The AGC functions as a sensitivity adjustment unit that adjusts the sensitivity during shooting, adjusting the gain of an amplifier that amplifies the input image signal so that the signal level of the image signal falls within an appropriate range. The sensitivity during shooting may be, for example, ISO sensitivity (International Organization for Standardization (ISO)).

[0119] When capturing a still image or a video, image data for each RGB pixel (mosaic image data) output via the image sensor 110 and the AFE 180 is input to and temporarily stored in the memory 130. Note that when the image sensor 110 is a CMOS image sensor, the AFE 180 is often built into the image sensor 110.

[0120] The processor 120 also functions as a digital signal processing unit that performs various types of digital signal processing on image data temporarily stored in the memory 130. That is, the processor 120 performs digital signal processing such as offset processing, gain control processing including sensitivity correction, gamma correction processing, demosaic processing, RGB / YCrCb conversion processing, etc. on the image data input via the AFE 180, and stores the image data after digital signal processing back in the memory 130. Note that, for example, in the case of an image sensor 110 consisting of RGB three-color filters, demosaic processing is a process of calculating all RGB color information for each pixel from a mosaic image consisting of RGB, and generating synchronized image data of three RGB planes from the mosaic data (dot-sequential RGB data). Demosaic processing is also called demosaicing processing or synchronization processing.

[0121] The RGB / YCrCb conversion process is a process for converting the synchronized RGB data into luminance data (Y) and color difference data (Cr, Cb).

[0122] Furthermore, when recording still images or moving images, the processor 120 compresses the uncompressed luminance data Y and color difference data Cb, Cr temporarily stored in the RAM of the memory 130. In the case of still images, the data is compressed in, for example, JPEG (Joint Photographic Coding Experts Group) format, and in the case of moving images, the data is compressed in, for example, H.264 format. The compressed image data is recorded in the flash memory of the memory 130. In addition, in playback mode, the processor 120 reads the compressed image data from the flash memory of the memory 130, decompresses the read image data, generates uncompressed image data, and displays it on the LCD 118 or the like via the display driver 140.

[0123] When displaying a live view image on the EVF 112 or the LCD 118, the processor 120 outputs a digital image signal that is captured and digitally processed at a predetermined frame rate to the display driver 140. The frame rate may be, for example, 30 frames per second (fps) or 60 fps.

[0124] The display driver 140 converts the input time-series digital image signals into a signal format for display and sequentially outputs them to the EVF 112 or the LCD 118. As a result, the captured image is displayed on the EVF 112 or the LCD 118 in real time.

[0125] The shutter release button 105 is a shooting instruction unit for inputting shooting instructions for still images and videos, and is configured as a two-stroke switch consisting of a so-called "half-press" (S1 press) and a "full press" (S2 press).

[0126] When the shutter release button 105 is "half-pressed," an S1_ON signal is output, and when it is further pressed from the "half-press" to the "full press," an S2_ON signal is output. In still image shooting mode, when the S1_ON signal is output, the processor 120 executes shooting preparation processing such as autofocus (AF) control and auto exposure (AE) control, and when the S2_ON signal is output, it executes still image shooting processing and recording processing.

[0127] When performing AF control, the processor 120 calculates values ​​necessary for AF control based on a digital image signal. In the case of so-called contrast AF, for example, the processor 120 calculates an integrated value (focus evaluation value) of the high-frequency components of the G signal within a predetermined AF area. The processor 120 moves the focus lens included in the lens group of the interchangeable lens 102 to a position where the focus evaluation value is maximized during AF control, i.e., where the contrast is maximized. Note that AF is not limited to contrast AF. For example, phase difference AF may be performed, in which the amount of defocus is detected based on pixel data from phase difference detection pixels provided in the image sensor, and the focus lens is moved so that this defocus amount becomes zero.

[0128] When performing AE control, the processor 120 detects the brightness of the subject (subject luminance) and calculates an exposure value, which is a numerical value required for AE control corresponding to the subject luminance. The exposure value is also called an EV value. The processor 120 can determine the F-number (aperture value), shutter speed, and ISO sensitivity from a predetermined program chart based on the calculated EV value, and perform AE control.

[0129] It goes without saying that AF control and AE control are performed automatically when the auto mode is set by the operation unit 150, and that AF control and AE control are not performed when the manual mode is set.

[0130] In addition, in the video shooting mode, when the shutter release button 105 is pressed all the way down and an S2_ON signal is output, the camera body 104 enters a video recording mode in which video recording begins, and performs image processing and recording processing for the video. After that, when the shutter release button 105 is pressed all the way down again and an S2_ON signal is output, the camera body 104 enters a standby state and temporarily suspends the video recording processing.

[0131] As shown in FIG. 12 , the infrared camera 200 includes an image sensor 210 , an AFE 220 , and an input / output interface 230 .

[0132] The image sensor 210 receives light in a wavelength range including the infrared region and outputs a captured image. The image sensor 210 is configured as a CMOS or CCD image sensor. The image sensor 210 has a microlens arranged on each of a plurality of pixels, each of which is configured as a photodiode arranged two-dimensionally in the x and y directions.

[0133] An optical image of the subject formed on the light receiving surface of the image sensor 210 by the imaging optical system of the infrared lens 202 is converted into an electrical signal by the image sensor 210. Charges corresponding to the amount of incident light are accumulated in each pixel of the image sensor 210, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 210 as an image signal.

[0134] The AFE 220 performs various types of analog signal processing on analog image signals obtained by capturing an image of a subject with the image sensor 210, and converts the processed image signals into digital image signals.

[0135] The value of each pixel in the infrared image captured by the infrared camera 200 indicates the intensity of infrared light, which is proportional to the temperature of the subject. The frame rate of the infrared camera 200 may be the same as the frame rate of the visible light camera 100. The frame rate of the infrared camera 200 may also be lower than the frame rate of the visible light camera 100.

[0136] When the visible light camera 100 configured as described above is powered on, the visible light camera 100 and the infrared camera 200 enter a shooting standby state. In the shooting standby state, the visible light camera 100 and the infrared camera 200 begin shooting video. The shot video is displayed as a live view image on the EVF 112 or LCD 118.

[0137] The user can determine the composition, confirm the subject they want to photograph, and set the photographing conditions by viewing the live view image displayed on the EVF 112 or the LCD 118. An example of the density change process for the visible light image that is applied to the display of the live view image when the infrared camera 200 is used will be described below.

[0138] [Example of Density Change Processing of Visible Light Image in Live View Display] FIG. 14 is a flowchart showing an example of density change processing of a visible light image in live view display.

[0139] In step ST11, the processor 120 determines whether or not the infrared camera 200 is attached. For example, the processor 120 checks whether or not there is an electrical connection between the visible light camera 100 and the infrared camera 200, and determines whether or not to use the infrared camera 200.

[0140] If the determination result in step ST11 is Yes, the processor 4 proceeds to step ST12. In step ST12, the processor 120 acquires the visible light image VSI captured by the image sensor 110 and the infrared image IRI captured by the image sensor 210.

[0141] In step ST13, the processor 120 acquires temperature information from the infrared image IRI.

[0142] In step ST14, processor 120 acquires information about a set temperature range to be applied to detecting the area of ​​the main subject included in infrared image IRI. This set temperature range may be set by the user before capturing the image, for example, using the method described with reference to FIG. 4. Processor 120 distinguishes between pixels within the set temperature range and pixels outside the set temperature range based on the temperature information of each pixel in infrared image IRI, and detects the area of ​​pixels within the set temperature range as the area of ​​the main subject.

[0143] In step ST15, the processor 120 determines whether or not the user has set a different density (user setting) from the default setting when displaying a live view image.

[0144] If the determination result in step ST15 is No, that is, if the user does not particularly specify a set concentration, the default setting is applied and the process proceeds to step ST16.

[0145] In step ST16, processor 120 determines, for each pixel in visible light image VSI, whether the pixel corresponds to a pixel in infrared image IRI that is within a set temperature range. Processor 120 may extract pixels from visible light image VSI that correspond to pixels within the set temperature range based on the correspondence between pixel positions in infrared image IRI and visible light image VSI. Processor 120 may determine that a region of pixels in visible light image VSI that corresponds to pixels within the set temperature range in infrared image IRI is a region of the main subject in visible light image VSI. Similarly, processor 120 may determine that a region of pixels in visible light image VSI that corresponds to pixels outside the set temperature range in infrared image IRI is a region other than the main subject in visible light image VSI.

[0146] If the determination result in step ST16 is Yes, that is, if the pixel corresponds to a pixel within the set temperature range in the infrared image IRI and belongs to the area of ​​the main subject in the visible light image VSI, the processor 120 proceeds to step ST19.

[0147] In step ST19, the processor 120 does not perform density conversion on the pixel (does not change the pixel value), and leaves the density as is. After step ST19, the processor 120 proceeds to step ST23.

[0148] On the other hand, if the judgment result of step ST16 is No, that is, if the pixel corresponds to a pixel outside the set temperature range in the infrared image IRI and belongs to an area other than the main subject in the visible light image VSI, the processor 120 proceeds to step ST17 and evaluates the density of the area of ​​the main subject.

[0149] In step ST17, processor 120 evaluates the density (brightness) of the area of ​​the main subject in visible light image VSI and determines whether it is brighter than a threshold. For example, processor 120 determines whether the average density of the main subject is greater than a threshold of 127.5. The average density here is an example of an index for evaluating the density of the area of ​​the main subject.

[0150] If the determination result in step ST17 is Yes, that is, if the density of the area of ​​the main subject is brighter than the threshold value, the processor 120 proceeds to step ST20.

[0151] In step ST20, the processor 120 sets the gamma value of the gamma conversion to γ=a according to the default setting, where a is a value smaller than 1, and may be, for example, 0.4. After step ST20, the processor 120 proceeds to step ST23.

[0152] On the other hand, if the determination result in step ST17 is No, that is, if the density of the main subject area is darker than the threshold value, the process proceeds to step ST21.

[0153] In step ST21, the processor 120 sets the gamma value of the gamma conversion to γ=b according to the default setting, where b is a value greater than 1 and may be, for example, 2.5. After step ST21, the processor 120 proceeds to step ST23.

[0154] If the determination result in step ST15 is Yes, that is, if the user has specified a set concentration, the processor 120 proceeds to step ST18.

[0155] In step ST18, the processor 120 determines for each pixel in the visible light image VSI whether or not the pixel corresponds to a pixel outside the set temperature range in the infrared image IRI.

[0156] If the determination result in step ST18 is No, that is, if the pixel belongs to the area of ​​the main subject in the visible light image VSI, the processor 120 proceeds to step ST19.

[0157] On the other hand, if the determination result in step ST18 is a Yes determination, that is, if the pixel belongs to the area of ​​the main subject in the visible light image VSI, the processor 120 proceeds to step ST22.

[0158] In step ST22, the processor 120 sets the gamma value for gamma conversion to a user-specified value γ=c, where c is, for example, a gamma value corresponding to the brightness level specified by the user on the image density setting screen 30. After step ST22, the processor 120 proceeds to step ST23.

[0159] In step ST23, density conversion is performed by applying the gamma value determined in any one of steps ST19 to ST22. Steps ST13 to ST23 are executed for each pixel of the visible light image VSI.

[0160] In step ST24, the processor 120 determines whether or not the processing of the final pixel in the visible light image VSI has been completed. If the determination result in step ST24 is No, the processor 120 changes the target pixel to the next pixel, returns to step ST13, and repeats steps ST13 to ST22.

[0161] When the process has been completed for all pixels in the visible light image VSI and the determination result in step ST24 is Yes, the processor 120 ends the flow chart of FIG.

[0162] In this way, a display image DSI is generated as a live view image, and the display image DSI is displayed on the EVF 112 or the LCD 118. The process shown in the flowchart of Fig. 14 may be executed for each frame of the live view image.

[0163] Also, if the judgment result of step ST11 is No, the processor 120 may terminate the flowchart of Figure 14 and proceed to a processing flow in which a live view image is displayed by applying a known display control method to a visible light image without using an infrared image.

[0164] The image capturing system 10 is preferably configured to allow a user to switch between a first display method, which uses information from the infrared image IRI to change the density of areas other than the main subject in the visible light image VSI, and a second display method, which displays the visible light image VSI (before the density change) without changing the density. For example, the image capturing system 10 may be configured so that a user can select the first display method or the second display method from a menu screen on which various settings of the image capturing system 10 are made.

[0165] [Regarding the program for causing a computer to realize the functions of the display control device 2] A program for causing a computer to realize some or all of the processing functions of the display control device 2 described in the above embodiment can be recorded on a computer-readable medium that is a tangible, non-transitory information storage medium, such as an optical disk, a magnetic disk, a semiconductor memory, or other such medium, and the program can be provided through this information storage medium. Furthermore, instead of providing the program by storing it on such a tangible, non-transitory information storage medium, it is also possible to provide a program signal as a download service using a telecommunications line such as the Internet.

[0166] In addition, some or all of the processing functions of the display control device 2 described in the above embodiment may be realized by cloud computing, and it is also possible to provide a service that provides processing functions as a SaaS (Software as a Service) service.

[0167] Advantages of this Embodiment The display control device 2 and the imaging system 10 according to the embodiment of the present disclosure described above have the following advantages.

[0168] [1] It is possible to use the infrared image IRI to clearly present the main subject in the visible light image VSI to the user.

[0169] [2] The user can set a temperature range that will be the temperature condition when detecting the area of ​​the main subject from the infrared image IRI. This makes it easier for the user to find and display only the subject they want to photograph.

[0170] [3] By following the default setting for display density and setting the gamma value of the area other than the main subject to γ ​​= a < 1 or γ = b > 1 depending on the density of the main subject in the visible light image VSI, the density difference between the area of ​​the main subject and the area other than the main subject can be increased.

[0171] [4] In addition to the default settings, as explained in FIG. 8, the user can freely set the gradation characteristics of the display, so that each user can set a density that is easy to see.

[0172] [5] The display of the live view image can be switched to the display of the visible light image VSI before the density change, or can be returned to the display of the live view image after the density change, via a user interface such as a menu screen. By enabling such display switching, the user can, as needed, change to the display of the visible light image VSI before the density change, allowing the user to, for example, capture while adjusting the white balance of the actual visible light image VSI.

[0173] 14, the density of the area of ​​the main subject in the visible light image VSI is left unchanged, but the display density of the area of ​​the main subject may also be changed by modifying the gradation characteristics. By setting a gamma value of 1 or greater for one of the gradation characteristics applied to the area of ​​the main subject in the visible light image VSI and the gradation characteristics applied to areas other than the main subject, and setting a gamma value of less than 1 for the other, the display density may be changed between the area of ​​the main subject and the area other than the main subject.

[0174] [Variation 2] The manner in which the infrared camera 200 is attached to the visible light camera 100 is not limited to the examples shown in Figures 10 and 11. For example, the infrared camera 200 may be fixed to a camera grip (not shown) and attached to the visible light camera 100 via the camera grip. Also, for example, the visible light camera 100 and the infrared camera 200 may be fixed to a common camera platform, or each may be fixed to a separate camera platform.

[0175] [Variation 3] The technology of the present disclosure is not limited to the imaging system 10 illustrated in FIGS. 10 and 11 , but can be applied to various system configurations, such as a surveillance camera system, an in-vehicle camera system, a drone imaging system, a wearable camera system, a broadcast video imaging system, or an imaging system that combines a camera-equipped mobile terminal such as a smartphone with an infrared camera.

[0176] [Others] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the technical idea of ​​the present disclosure.

[0177] DESCRIPTION OF SYMBOLS 2...Display control device 4...Processor 6...Storage device 10...Photographing system 12...Visible light image acquisition section 14...Infrared image acquisition section 16...Display section 18...Operation section 30...Image density setting screen 32...Brightness adjustment bar 34...Image display area 36...Mark 38...Set button 40...Infrared image reception section 42...Temperature information conversion section 44...Temperature range setting section 46...Main subject area detection section 50...Visible light image reception section 51...Corresponding area detection section 52...Density evaluation section 54...Gradation characteristics control section 56...Density conversion section 58...Display image output section 60...Main subject area gradation characteristics change section 62...Background area gradation characteristics change section 64...Main subject area density conversion section 66...Background area density conversion section 100...Visible light camera 102...Interchangeable lens 104...Camera body 105...Shutter release button 106...Shutter speed dial 107...Exposure compensation dial 108...Power lever 109...Hot shoe 110...Image sensor 112...EVF 114...MENU / OK key 116...Cross key 117...Playback button 118...LCD 120...Processor 122...Communication cable 130...Memory 140...Display driver 150...Operation unit 160...Input / output interface 170...Sensor driver 180...AFE 200...Infrared camera 202...Infrared lens 204...Camera body 210...Image sensor 220...AFE 230...Input / output interface AI...Angle of view AV...Angle of view IRI, IRI1...Infrared image VSI, VSI1, VSI2, VSI3...Visible light image DSI, DSI1, DSI2, DSI3...Image for display MS1...Main subject MSv1, MSv2, MSv3, MSv4...Main subject TC1...Main subject area gradation characteristics TC2...Background area gradation characteristics F8A...Left diagram F8B...Center diagram F8C...Right diagram F12A...Left diagram F12B...Center diagram F12C...Right diagram ST1 to ST6...Steps of display control method ST11 to ST24...Steps of method for changing display density of live view image

Claims

1. A visible light image is obtained by capturing an image in a wavelength range including the visible light range, Acquire an infrared image obtained by capturing an image in a wavelength range including the infrared range, displaying the visible light image; Detecting a region of a main subject based on temperature information of the subject contained in the infrared image; a processor that changes gradation characteristics in display of at least a first region, which is a region of the visible light image corresponding to the region, and a second region, which is a region other than the main subject in the visible light image, to make the gradation characteristics different between the first region and the second region; the processor changes the gradation characteristics of the second area in accordance with the brightness distribution of the first area so that a difference in brightness between the first area and the second area in display becomes large. Display control device.

2. The processor: changing the gradation characteristics of the second region based on the temperature information and displaying the visible light image; The display control device according to claim 1 .

3. The processor: changing the gradation characteristics of the second region based on the index value related to the brightness of the first region; The display control device according to claim 1 .

4. The processor, comparing the index value with a threshold value, and when the brightness of the first region is darker than the brightness corresponding to the threshold value, changing the gradation characteristics of the second region so as to brighten the display of the second region, and when the brightness of the first region is brighter than the brightness corresponding to the threshold value, changing the gradation characteristics of the second region so as to darken the display of the second region; The display control device according to claim 3 .

5. The processor: Accepting a designation of a gradation setting to be applied to the display of the second region; changing the gradation characteristics of the second region in accordance with the specified gradation setting; The display control device according to claim 1 .

6. The processor: detecting the area of ​​the main subject according to a preset temperature condition; The display control device according to claim 1 .

7. The processor: detecting an area in the infrared image that satisfies a specific temperature condition based on the temperature information as the area of ​​the main subject; The display control device according to claim 1 .

8. The processor: detecting an area in the infrared image showing a temperature between 30°C and 42°C based on the temperature information as the area of ​​the main subject; The display control device according to claim 7 .

9. The processor: Accepting designation of a set temperature range to be applied to detection of the area of ​​the main subject; detecting the area in the infrared image within the specified temperature range as the area of ​​the main subject; The display control device according to claim 1 .

10. The processor: changing the gradation characteristics by changing a gamma value of the gradation characteristics in the visible light image; The display control device according to claim 1 .

11. one of the gradation characteristics applied to the first region and the gradation characteristics applied to the second region in the visible light image has a gamma value of 1 or more, and the other has a gamma value of less than 1; The display control device according to claim 10.

12. a first image acquisition unit that acquires a visible light image obtained by capturing an image in a wavelength range including the visible light range; a second image acquisition unit that acquires an infrared image obtained by capturing an image in a wavelength range including an infrared region; a display unit that displays the visible light image; a processor that detects a region of a main subject based on temperature information of the subject contained in the infrared image, and changes gradation characteristics in display of at least a first region that is a region of the visible light image corresponding to the detected region and a second region that is a region other than the main subject in the visible light image, thereby differentiating the gradation characteristics between the first region and the second region; Equipped with the processor changes the gradation characteristics of the second area in accordance with the brightness distribution of the first area so that a difference in brightness between the first area and the second area in display becomes large. Shooting system.

13. the first image acquisition unit includes a visible light camera; the second image acquisition unit includes an infrared camera; The imaging system according to claim 12.

14. the visible light camera includes the display unit and the processor; The imaging system according to claim 13.

15. The infrared camera is used in connection with the visible light camera. The imaging system according to claim 13.

16. A display control method executed by a processor, comprising: the processor: A visible light image obtained by capturing an image in a wavelength range including the visible light range and an infrared image obtained by capturing an image in a wavelength range including the infrared range are acquired, displaying the visible light image; Detecting a region of a main subject based on temperature information of the subject contained in the infrared image; performing a process of changing gradation characteristics in displaying at least a second area of ​​a first area of ​​the visible light image corresponding to the area and a second area of ​​the visible light image other than the main subject, and making the gradation characteristics of the first area and the second area different from each other; the processor changes a gradation characteristic of the second area in accordance with a brightness distribution of the first area so that a difference in brightness between the first area and the second area in display increases. Display control method.

17. On the computer, a function of acquiring a visible light image obtained by imaging in a wavelength range including the visible light range and an infrared image obtained by imaging in a wavelength range including the infrared range; a function of displaying the visible light image; a function of detecting a region of a main subject based on temperature information of the subject contained in the infrared image; a function of changing gradation characteristics in display of at least a first region, which is a region of the visible light image corresponding to the region, and a second region, which is a region other than the main subject in the visible light image, to make the gradation characteristics of the first region and the second region different from each other; To achieve this, causing the computer to realize a function of changing the gradation characteristics of the second area in accordance with the brightness distribution of the first area so that the difference in brightness between the first area and the second area in display becomes larger; program.

18. A non-transitory computer-readable recording medium having the program according to claim 17 recorded thereon.