Information processing apparatus, imaging apparatus, control method, and program

Abstract

To determine exposure settings for acquiring a visible light image and a near infrared image with less deviation in resolution feel.SOLUTION: An information processing apparatus determines exposure settings of an imaging apparatus that photographs a subject for rays of light in two types of wavelength regions, and the information processing apparatus has: first acquisition means that acquires results of photometry of the subject for the light in a first wavelength region and the light in a second wavelength region including a wavelength longer than that of the first wavelength region; first determination means that determines a first exposure setting for photographing the subject for the light in the first wavelength region on the basis of the result of photometry for the light in the first wavelength region; and second determination means that determines a second exposure setting for photographing the subject for the light in the second wavelength region on the basis of the result of photometry for the light in the second wavelength region and the first exposure setting. The exposure settings include f numbers. The second determination means determines, for the second exposure setting, a second f number closer to an opening side than a first f number determined by the first determination means.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an information processing device, an imaging device, a control method, and a program, and more particularly to a technique for capturing a visible light image and a near-infrared light image of a subject. [Background technology]

[0002] In the field of machine vision used in factory automation and the like, near-infrared images are taken in addition to visible light images and are used to grasp the state of a subject. Due to the property of near-infrared light easily penetrating materials, near-infrared images can be used, for example, to determine the freshness of agricultural products, detect foreign matter in food, and detect defects in industrial products. On the other hand, visible light images can obtain accurate shape and color information, so various states of a subject can be detected / determined by combining these images. Therefore, in the field of machine vision, there are examples in which imaging devices capable of taking visible light images and near-infrared images are adopted.

[0003] However, the axial chromatic aberration that occurs through the optical system is different between visible light and near-infrared light. Therefore, when an image is captured with the focus adjusted based on visible light, the depth of field and the focus state (focus position) of the subject may differ between the visible light image and the near-infrared light image. If there is such a discrepancy between the visible light image and the near-infrared light image, the reliability of an inspection using the image group, for example, will decrease. For this reason, there is also an imaging device that combines and displays a visible light image and a near-infrared light image in a manner that allows the difference in the focus state of each image to be confirmed (Patent Document 1). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-011575 Summary of the Invention [Problem to be solved by the invention]

[0005] The difference in focus between a visible light image and a near-infrared image is not limited to axial chromatic aberration caused by the optical system. Diffraction caused by the aperture is also a factor. When a visible light image and a near-infrared image are taken with the same aperture value, the near-infrared light, which has a longer wavelength, is more spread out due to diffraction and focused on the image sensor, resulting in an image with a lower sense of resolution.

[0006] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an information processing device, an imaging device, a control method, and a program that determine exposure settings for acquiring visible light images and near-infrared light images with little discrepancy in resolution. [Means for solving the problem]

[0007] In order to achieve the above-mentioned object, the information processing device of the present invention is an information processing device that determines exposure settings of an imaging device that photographs a subject for light in two wavelength ranges, and includes a first acquisition means that acquires photometric results of the subject for each of light in a first wavelength range and light in a second wavelength range that includes a wavelength longer than the first wavelength range, a first determination means that determines a first exposure setting for photographing the subject for light in the first wavelength range based on the photometric results of the light in the first wavelength range, and a second determination means that determines a second exposure setting for photographing the subject for light in the second wavelength range based on the photometric results of the light in the second wavelength range and the first exposure setting, wherein the exposure settings include an aperture value, and the second determination means determines a second aperture value for the second exposure setting that is more open than the first aperture value determined by the first determination means. Effect of the Invention

[0008] With this configuration, according to the present invention, it is possible to determine exposure settings for acquiring a visible light image and a near-infrared light image with little difference in perceived resolution. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram illustrating a configuration of an image capturing apparatus 100 according to a first embodiment of the present invention. [Diagram 2] FIG. 1 is a diagram illustrating an image capturing unit 304 according to the first embodiment of the present invention. [Diagram 3] FIG. 13 is a diagram illustrating the correspondence relationship between the Tv value and the shutter speed, between the Av value and the aperture value, and between the ISO sensitivity and the Ev value correction step number according to the embodiment and the modified example of the present invention. [Figure 4] FIG. 13 is a diagram illustrating the relationship between each parameter of the exposure setting according to the embodiment and the modified example of the present invention; [Diagram 5] FIG. 13 is a diagram illustrating an example of the Airy disk diameter for each aperture value for each wavelength according to the embodiment and the modified example of the present invention; [Figure 6] 1 is a flowchart illustrating an example of a photographing process executed in the image capturing device 100 according to the first embodiment of the present invention. [Figure 7] 1 is a flowchart illustrating a determination process executed in the image capture device 100 according to the first embodiment of the present invention. [Figure 8] FIG. 1 is a block diagram illustrating the configuration of an image capture device 100 according to a third modification and a second embodiment of the present invention. [Figure 9] FIG. 11 is a diagram for explaining the spectral sensitivity characteristics of the image capturing unit 801 and the image capturing unit 802 according to the third modification and the second embodiment of the present invention; [Figure 10] 11 is a flowchart illustrating a photographing process executed in the image capturing device 100 according to the second embodiment of the present invention. [Figure 11] 11 is a flowchart illustrating a determination process executed in the image capture device 100 according to the second embodiment of the present invention. [Figure 12] FIG. 13 is a diagram illustrating the spectral characteristics of a light-emitting device according to a fourth modified example of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] [Embodiment 1] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] In the embodiment described below, an example in which the present invention is applied to an image capturing device capable of capturing visible light and near-infrared light as an example of an information processing device will be described. However, the present invention can be applied to any device capable of determining exposure settings to be set when capturing each image, in an image capturing device capable of capturing an image of a subject using light in two wavelength ranges.

[0012] Configuration of the imaging device 1 is a block diagram showing the functional configuration of an image capturing apparatus 100 according to an embodiment of the present invention. In this embodiment, the image capturing apparatus 100 is of a lens-interchangeable type, and will be described as being composed of a lens barrel 200 and a main body unit 300.

[0013] The lens barrel 200 is an imaging optical system configured to transmit wavelengths in the visible light region and the near-infrared light region. As shown in the figure, a focus adjustment lens 203 and an aperture 205 are provided on the optical axis of the lens barrel 200. The focus adjustment lens 203 is configured to be driven back and forth along the optical axis by a lens drive unit 202. By changing the position of the focus adjustment lens 203 on the optical axis, it is possible to adjust the focus state of an optical image formed on an imaging unit 304 of a main body unit 300 described later. In addition, the aperture 205 is configured so that the opening and closing state of the aperture can be controlled by the aperture drive unit 204. By changing the opening and closing state of the aperture 205, it is possible to adjust the amount of light incident on the main body unit 300 and the depth of field. Each unit of the lens barrel 200 is controlled by a lens control unit 201. The lens control unit 201 controls the lens driving unit 202 and the aperture driving unit 204 to perform drive control based on information for controlling the states of the focusing lens 203 and the aperture 205 received from the main body unit 300 via the electrical contacts 206 .

[0014] The main body 300 is configured to be able to capture light incident via the lens barrel 200 and record the image.

[0015] The CPU 301 is a control device that controls the operation of each block included in the main body 300. The CPU 301 can control the operation of each block by reading out a control program for each block stored in the storage device 302, expanding the program in the memory 303, and executing the program.

[0016] The storage device 302 is a non-volatile memory such as a flash memory. The storage device 302 is a device capable of permanently storing data, and can store image data obtained by shooting in addition to the control programs of each block. The memory 303 is a volatile memory such as a DRAM. The memory 303 is a storage device with a faster data access speed than the storage device 302, and can be used for expanding the control programs, storing intermediate data output by the operation of each block, and the like.

[0017] The imaging unit 304 includes an imaging element such as a CMOS sensor or a CCD, captures light incident through the lens barrel 200, and outputs image data. More specifically, the imaging unit 304 photoelectrically converts an optical image formed on the imaging surface of the imaging unit 304 by the lens barrel 200, obtains an analog image signal, and A / D converts the analog image signal to output image data (digital image signal). As shown in the figure, the imaging unit 304 is configured to include a pixel array 3041, an A / D converter 3042, a timing generator 3043, and a transfer I / F 3044.

[0018] 2(a), the pixel array 3041 has pixels arranged in a grid pattern to form an imaging surface. Each pixel is equipped with a photoelectric conversion element, a transfer transistor, an amplification transistor, and a source follower transistor, and pixel signals can be read out line by line by connecting them to a common vertical wiring.

[0019] As described above, the imaging device 100 of this embodiment is configured to be able to capture an image of a subject using both visible light (wavelengths of 380 nm to 780 nm) and near-infrared light (for example, wavelengths of 780 nm to 1000 nm). In order to capture these lights with one imaging element in the main body 300, a filter different from that of adjacent pixels is applied to each pixel of the pixel array 3041 as shown in FIG. 2(a). In the example shown in the figure, three types of filters, red (R), green (G), and blue (B), are applied as visible light filters, and one type of filter, IR, is applied as near-infrared light filters. In this embodiment, the imaging element of the imaging unit 304 is described as a CMOS sensor. The spectral sensitivity characteristic of the CMOS sensor to near-infrared light is as shown in FIG. 2(b), and light with wavelengths of 780 nm to 1000 nm can be captured as near-infrared light. In the illustrated example, the peak band where the image sensor has high sensitivity to near-infrared light is around 830 nm, and hereinafter, 830 nm will be described as the representative wavelength of the near-infrared light region.

[0020] The A / D converter 3042 performs A / D conversion processing on the analog image signal read out from the pixel array 3041, and converts it into a digital image signal. The timing generator 3043 generates and outputs timing signals related to each operation of the image sensor. The timing signals are used, for example, to control the timing of resetting, accumulating, transferring, and reading out the charge of the pixel array 3041. The transfer I / F 3044 transfers the image data obtained by A / D conversion to a block (the correction unit 305 in the example of FIG. 1) that handles subsequent processing related to imaging / photography. The correction unit 305 corrects shading of pixel values, dark current components, etc. of the input image data.

[0021] The photometry unit 306 measures the luminance distribution of the subject and outputs the photometry results (photometry data). For example, image data obtained by imaging can be used for photometry, and the luminance distribution of the entire frame (image data) can be measured by integrating the luminance value for each arbitrary block. The photometry data is transferred to the shooting control unit 309 (described later) as information on the subject light amount (Light Value, hereinafter referred to as Lv value), and is used to determine the exposure amount (Exposure Value, hereinafter referred to as Ev value) and exposure settings.

[0022] The focusing unit 307 generates information for controlling the focus state of the subject in the optical image formed on the imaging surface. The information is, for example, information on the image shift amount (defocus amount) related to the image estimated as the subject, and is transmitted to the optical system control unit 311.

[0023] The distance measuring unit 308 measures the subject distance (distance measurement). The distance measuring unit 308 can use a distance measuring device such as LiDAR, and can obtain the depth distribution of the subject included in the imaging range. The distance measuring unit 308 obtains the distance between the subject and the imaging surface as the subject distance based on the distance measurement result, and outputs it to the shooting control unit 309.

[0024] The shooting control unit 309 determines exposure settings (Av value corresponding to aperture value, Tv value corresponding to shutter speed, and ISO sensitivity) for shooting a subject. In the imaging device 100 of this embodiment, visible light and near-infrared light are shot of the subject in a time-division manner, so the shooting control unit 309 determines exposure settings for each of visible light and near-infrared light. The shooting control unit 309 outputs control instructions related to exposure to the imaging control unit 310 and the optical system control unit 311 based on the determined exposure settings.

[0025] The imaging control unit 310 controls the operation of the imaging unit 304. More specifically, the imaging control unit 310 sets the driving of the imaging unit 304 based on the input control instruction related to exposure, and also transfers the readout timing (synchronization signal). In the main body unit 300 of this embodiment, the imaging control unit 310 is configured to be able to operate independently based on the control instruction from the shooting control unit 309 so that it can operate without using the calculation resources of the CPU 301.

[0026] The optical system control unit 311 transmits information for controlling the state of the lens barrel 200 to the lens control unit 201 via the electrical contact 312 (and the electrical contact 206). When the optical system control unit 311 is supplied with information on the defocus amount from the focusing unit 307, it determines the drive amount (lens movement amount) of the focusing lens 203 and transmits it to the lens control unit 201. When the optical system control unit 311 is supplied with information on the exposure setting from the shooting control unit 309, it transmits the Av value and (if necessary) information on the focusing position to the lens control unit 201. At this time, the lens control unit 201 can perform processing such as converting the Av value into the drive amount of the aperture driving unit 204 (opening amount of the aperture 205) and converting the information on the focusing position into the drive amount of the lens driving unit 202 (movement amount of the focusing lens 203). When the lens control unit 201 receives information related to the state control of the lens barrel 200 in this manner, it transmits the necessary information to the lens driving unit 202 and the aperture driving unit 204 to control the states of the focus adjustment lens 203 and the aperture 205.

[0027] The electrical contacts 312 and 206 are an interface for transmitting and receiving information between the lens barrel 200 and the main body 300. When the lens barrel 200 is attached to the main body 300, the electrical contacts 312 and 206 are electrically connected, enabling information transmission and reception. In this embodiment, the interface between the lens barrel 200 and the main body 300 is described as an electrical contact, but it may also be serial communication, for example.

[0028] The codec 313 performs compression and decompression processing of image data. When shooting, the codec 313 performs compression processing on image data obtained by shooting based on information of a predetermined recording format. When playing back image data, the codec 313 performs decompression processing on image data recorded in, for example, the storage device 302 based on information of a predetermined recording format. The external communication unit 314 controls the external output of image data obtained by shooting. In a mode in which image data captured by the imaging device 100 is used in real time as machine vision, the external communication unit 314 outputs image data to an external device via the communication I / F 315. In this manner, in a mode in which image data is not recorded in the storage device 302, the codec 313 can be configured to perform only compression. The communication I / F 315 is an interface for transmitting and receiving information between the main body unit 300 and an external device. Communication with the external device may be performed either wired or wirelessly.

[0029] Overview of Exposure Setting An overview of the determination of exposure settings performed by the image capture control unit 309 in the main body unit 300 configured as above will be described below. As described above, in the imaging device 100 of this embodiment, in order to obtain an image of a subject captured using visible light (visible light image) and an image of a subject captured using near-infrared light (near-infrared light), the image capture control unit 309 determines exposure settings for each of them.

[0030] Prior to capturing each image, the photometry unit 306 performs photometry for each of visible light and near-infrared light, and outputs the Lv value to the capture control unit 309. In the imaging device 100 of this embodiment, capture is performed using an image sensor to which R, G, B, and IR filters are applied, as shown in Fig. 2(a). Therefore, the photometry unit 306 can obtain an Lv value (first Lv value) related to visible light and an Lv value (second Lv value) related to near-infrared light based on block integral values ​​of the luminance values ​​of each of the RGB pixels and IR pixels in the subject area from image data obtained in, for example, preliminary capture.

[0031] In order to capture a visible light image and a near-infrared light image capturing the subject in a suitable focused state, the distance measuring unit 308 acquires information indicating the distribution of the subject in the depth direction. The information may include a representative value of the subject distance, the subject distance of each pixel in the subject area, and the depth length of the subject (thickness in the depth direction).

[0032] The shooting control unit 309 first determines the exposure setting for visible light. In general, the exposure setting is determined by setting the Lv value as the target exposure value (Ev value) and selecting the Tv value, Av value, and ISO sensitivity so as to satisfy a predetermined relationship. Here, the Ev value, Tv value, and Av value are as follows: Ev value = Tv value + Av value The following relationship holds. The ISO sensitivity corresponds to the Ev value correction step number, and the above relationship can be expressed by taking into account the correction step number (ISO correction step number). Ev value = Tv value + Av value - ISO correction step Here, the Tv value and shutter speed, the Av value and aperture value, and the ISO sensitivity and correction step number have the corresponding relationships as shown in Fig. 3(a), (b), and (c). That is, the standard Ev value (Ev=0) is defined as the brightness of the subject that provides suitable exposure when photographed with an ISO sensitivity of 100, an aperture value of F1.0, and a shutter speed of 1 second. Therefore, for example, the Ev value that provides suitable exposure for a combination of Tv value and Av value at ISO 100 can be determined based on a table such as that shown in Fig. 4.

[0033] In visible light shooting, since the effect of diffraction can become significant when the aperture value is larger than F22 (the aperture diameter becomes smaller), an upper limit of the aperture value is set in advance in the exposure setting for visible light to avoid the occurrence of the effect of diffraction. That is, in determining the exposure setting for visible light, the shooting control unit 309 determines the Av value so that it is smaller than a predetermined upper limit aperture value (opener than a predetermined aperture diameter), and then determines the Tv value and ISO sensitivity.

[0034] On the other hand, visible light and near-infrared light are affected by diffraction differently even if they pass through the aperture 205 with the same aperture diameter. More specifically, since the wavelength of near-infrared light is longer, the light flux passing through the aperture 205 with the same aperture diameter is affected by diffraction and is imaged more widely on the imaging surface. That is, when shooting is performed with the aperture 205 with the same aperture diameter, the near-infrared light image may have a lower sense of resolution than the visible light image. Therefore, the shooting control unit 309 determines the exposure setting for infrared light by referring to the exposure setting for visible light in addition to the second Lv value so as to equalize the degree of influence of diffraction between the visible light image and the near-infrared light image.

[0035] Evaluation of the effect of diffraction In the imaging device 100 of this embodiment, the Airy disk diameter of the light imaged on the imaging plane is used as an evaluation value to evaluate the effect of diffraction. The Airy disk refers to a disk-shaped region in which an image formed on the imaging plane when a point light source is imaged through the imaging optical system spreads out and does not become a single point due to the effect of diffraction through the aperture 205. In other words, the Airy disk diameter is an index (radius of the region) indicating the size of the disk-shaped region.

[0036] The airy disc diameter D is D=1.22×λ×F Here, λ is the wavelength of light, and F is the aperture value (F-number). Therefore, in order to make the effect of diffraction when capturing a near-infrared light image the same as when capturing a visible light image, it is necessary to adjust the aperture value of the exposure setting related to the near-infrared light according to the difference in wavelength between visible light and near-infrared light. More specifically, since near-infrared light has a longer wavelength than visible light, the capture control unit 309 sets the aperture value when capturing a near-infrared light image to be smaller than that when capturing a visible light image, that is, to be more open than when capturing a visible light image.

[0037] Therefore, the photographing control unit 309 derives an aperture value at which the Airy disk diameter is equal to or smaller than the Airy disk diameter at the time of photographing a visible light image in the exposure setting for near-infrared light, based on the aperture value determined for the exposure setting for visible light. Here, since the visible light region and the near-infrared light region each include various wavelengths, the photographing control unit 309 performs calculations using the representative wavelength of each wavelength region. In other words, the photographing control unit 309 determines the aperture value at the time of photographing a near-infrared light image so that the Airy disk diameter does not exceed the Airy disk diameter of visible light at the aperture value determined for photographing a visible light image.

[0038] The representative wavelength of the visible light region can be, for example, the reference wavelength of optics and optical instruments. The reference wavelength can be e-line (546.07 nm) or d-line (587.56 nm) as specified in ISO7944:1998. In this embodiment, the representative wavelength of the visible light region is the wavelength (546.07 nm) of e-line, which is a mercury (Hg) spectral line with a wavelength of 546.1 nm. The representative wavelength of the near-infrared light region is 830 nm, which is the peak wavelength of the peak band in the spectral sensitivity characteristic of the near-infrared light of the imaging element. The Airy disk diameter for each aperture value for each wavelength may be derived in advance, and may be stored in the storage device 302 as a table such as that shown in FIG. 5, and the shooting control unit 309 may refer to this to determine the aperture value when shooting a near-infrared light image.

[0039] In the following, for ease of understanding of the invention, the exposure setting related to visible light will be referred to as the "first exposure setting" and the exposure setting related to near-infrared light will be referred to as the "second exposure setting." The aperture value set in the exposure setting related to visible light (aperture value when a visible light image is captured) will be referred to as the "first aperture value," and the aperture value set in the exposure setting related to near-infrared light (aperture value when a near-infrared image is captured) will be referred to as the "second aperture value." The Av value corresponding to the first aperture value set in the exposure setting related to visible light will be referred to as the "first Av value," and the Av value corresponding to the second aperture value set in the exposure setting related to near-infrared light will be referred to as the "second Av value." The representative wavelength in the visible light region will be referred to as the "first wavelength," and the representative wavelength in the near-infrared region will be referred to as the "second wavelength."

[0040] That is, the photographing control unit 309 determines the first exposure setting based on the first Lv value, and then determines the second Av value based on the first Av value of the first exposure setting and the representative wavelength of each wavelength range. Then, the photographing control unit 309 determines other parameters (Tv value and ISO sensitivity) of the second exposure setting based on the second Av value and the second Lv value.

[0041] <Focus bracketing> In this way, the shooting control unit 309 determines an aperture value for the second exposure setting that is more open than the first exposure setting. On the other hand, when the aperture 205 is set to the open side, the depth of field is narrowed accordingly. That is, if the aperture value is adjusted so that the effect of diffraction is the same for the visible light image and the near-infrared light image, a difference in the focus state of the subject may occur between the visible light image and the near-infrared light image. For this reason, in the imaging device 100 of this embodiment, the near-infrared light image is acquired by focus bracketing shooting in which the images are shot sequentially with different focus positions. That is, since the second Av value is set to the open side more than the first Av value, the depth of field of the near-infrared light image obtained by one shooting becomes narrow, but by shooting multiple times with different focus positions, the subject captured in the depth of field of the visible light image is covered by multiple near-infrared light images.

[0042] Therefore, the shooting control unit 309 also determines the number of times of shooting for focus bracket shooting and the focus position set for each shooting (which can be converted into the driving amount of the actuator of the lens driving unit 202), and includes them in the second exposure setting. The number of times of shooting can be derived, for example, by dividing the range (distance) in the depth direction in which the subjects to be shot in focus are distributed by the range (distance) in the depth direction included in the depth of field at the second Av value (by rounding up) and casting it to an integer value.

[0043] Here, the depth of field can be calculated by summing the front depth of field and the rear depth of field calculated based on the subject distance (distance to the in-focus position) d of the in-focus subject, the second Av value, the focal length f of the imaging optical system, and the definition value r of the permissible circle of confusion in the imaging unit 304. The front depth of field is calculated by summing the front depth of field and the rear depth of field. (Front depth of field) = (r × (second aperture value) × d 2 ) / (f2 +r×(2nd aperture value)×d) The rear depth of field can be calculated as follows: (Rear depth of field) = (r × (2nd aperture value) × d 2 ) / (f 2 -r x (2nd aperture value) x d) Here, the definition value r of the permissible circle of confusion can be, for example, the pixel pitch of the pixel array 3041 of the imaging unit 304 or the Airy disk diameter at the second aperture value, whichever is larger. Also, the focal length f has the property of becoming larger as the wavelength becomes longer.

[0044] The focus position for each shooting may be determined based on a predetermined rule so that the entire range (target range) to be shot in focus falls within the depth of field of the multiple near-infrared light images. For example, when three shootings are performed as focus bracket shooting, the shooting control unit 309 can determine the focus position for each shooting so that the closest end of the depth of field for the first shooting coincides with the closest end of the target range, and the farthest end of the depth of field for the third shooting coincides with the farthest end of the target range. The focus position for the second shooting can be determined so that the focus adjustment lens 203 is located at the midpoint between the position of the focus adjustment lens 203 for the first shooting and the position of the focus adjustment lens 203 for the third shooting.

[0045] <Image Output> When the shooting control unit 309 determines the first exposure setting and the second exposure setting, this information is supplied to the shooting control unit 309 and the lens control unit 201, and the visible light image and the near-infrared light image are shot. From the viewpoint of use as a machine vision, it is preferable that these images are output in a manner that allows easy comparison, so the CPU 301 outputs the obtained visible light image and near-infrared light image in association with each other. As described above, the output may be output to an external device via the external communication unit 314, or may be recorded in the storage device 302 after compression processing by the codec 313. At this time, the multiple near-infrared light images obtained by focus bracket shooting may be converted into one composite image by depth stacking and output in consideration of convenience when comparing with the visible light image. The composite image can be generated, for example, by averaging multiple near-infrared light images, and can present a mode in which the depth of field is expanded more than that of a single near-infrared light image.

[0046] 《Shooting Processing》 Hereinafter, a specific photographing process for outputting a visible light image and a near-infrared light image of a subject in the imaging device 100 of this embodiment will be described with reference to the flowchart of FIG. 6. The process corresponding to the flowchart can be realized by the CPU 301 reading out a corresponding processing program stored in the storage device 302, for example, and executing the program by expanding it in the memory 303. This photographing process will be described as being started when it is detected that a photographing instruction for two types of light has been received via a user interface (not shown). It is assumed that photometry by the photometry unit 306 and distance measurement by the distance measurement unit 308 have been performed prior to the start of this photographing process. It is also assumed that the focus adjustment lens has been moved to a position where the subject is focused prior to the start of this photographing process.

[0047] In S601, under the control of the CPU 301, the shooting control unit 309 determines each parameter (including the first Av value) of the first exposure setting based on the input first Lv value.

[0048] In S602, under the control of the CPU 301, the shooting control unit 309 executes a determination process for determining each parameter of the second exposure setting based on the input second Lv value and the first Av value determined in S601.

[0049] Decision Processing Here, the determination process executed in this step will be described in detail with reference to the flowchart of FIG.

[0050] In S701, the shooting control unit 309 acquires the Airy disk diameter at the exposure setting of the visible light image based on the first aperture value and the first wavelength corresponding to the first Av value. The Airy disk diameter may be derived by calculation or may be acquired from a table or the like.

[0051] In S702, the shooting control unit 309 selects an aperture value for near-infrared light that results in an Airy Disk diameter equal to or smaller than the Airy Disk diameter derived in S701 as a target aperture value. The shooting control unit 309 also sets the Av value corresponding to the target aperture value as a target Av value.

[0052] In S703, the shooting control unit 309 derives (provisionally determines) a Tv value to be the second exposure setting based on the target Ev value set based on the second Lv value and the target Av value set in S702. As described above, the Tv value is Tv value = target Ev value - target Av value It can be derived as follows.

[0053] In S704, the imaging control unit 309 judges whether the Tv value derived in S703 is equal to or less than a predetermined upper limit. For example, if the main body unit 300 has a mechanical shutter mechanism, the upper limit of the Tv value can be set based on the performance of the mechanism. Alternatively, for example, if the sensor drive method of the imaging unit 304 is a rolling shutter method, the upper limit of the Tv value can be determined based on the charge readout speed. If the Tv value is equal to or less than the upper limit, the imaging control unit 309 transfers the process to S705, and if the Tv value exceeds the upper limit, the imaging control unit 309 transfers the process to S712.

[0054] In S705, the shooting control unit 309 judges whether the Tv value derived in S703 is equal to or greater than a predetermined lower limit. The lower limit of the Tv value can be set, for example, according to the magnitude of the movement of the subject. Here, if there is no movement of the subject, the lower limit can be set low (slower shutter speed), but it is set to a value that avoids a decrease in the throughput of the entire imaging device 100. If the shooting control unit 309 judges that the Tv value is equal to or greater than the lower limit, it moves the process to S706, and if it judges that the Tv value is below the lower limit, it moves the process to S707.

[0055] In S706, the shooting control unit 309 determines the target Av value as the second Av value.

[0056] On the other hand, if it is determined in S705 that the Tv value falls below the lower limit, the shooting control unit 309 updates the target Ev value to a value increased by 1 in S707. That is, the shooting control unit 309 increases the ISO correction step number by one step to increase sensitivity. That is, the shooting control unit 309 determines the ISO sensitivity one step higher as the second exposure setting.

[0057] In S708, the shooting control unit 309 judges whether the ISO sensitivity determined in S707 has reached the upper limit. The ISO sensitivity is changed by applying a gain to the analog image signal before conversion in the A / D converter 3042, or by applying a digital gain after conversion. More specifically, the ISO sensitivity is changed according to the range of the slope of the reference signal of the A / D converter 3042, the gain setting of the analog amplifier in the previous stage, and the gain setting of the digital gain in the subsequent stage. On the other hand, since the image quality required varies depending on the use of the image data obtained by shooting, the ISO sensitivity is provided with upper and lower limit values ​​according to the use, for example. If the shooting control unit 309 judges that the ISO sensitivity has reached the upper limit, the processing proceeds to S709. If the shooting control unit 309 judges that the ISO sensitivity has not reached the upper limit, the processing returns to S703, and the Tv value to be the second exposure setting for the updated target Ev value is derived again (provisionally determined).

[0058] In S709, the shooting control unit 309 updates the target Av value to a value decreased by 1. That is, the shooting control unit 309 updates the target Av value to an aperture value that opens up by one stop.

[0059] In S710, the shooting control unit 309 judges whether or not the target Av value updated in S709 has reached the lower limit of the Av value setting. Here, the lower limit of the Av value setting can be the (maximum) open state of the aperture 205. If the shooting control unit 309 judges that the updated target Av value has reached the lower limit of the setting, it transfers the process to S711. If the shooting control unit 309 judges that the updated target Av value has not reached the lower limit of the setting, it returns the process to S703, and re-derives (provisionally determines) the Tv value to be the second exposure setting based on the updated target Av value and the updated target Ev value.

[0060] In S711, the shooting control unit 309 determines the lower limit of the Av value setting as the second Av value.

[0061] On the other hand, if it is determined in S704 that the Tv value exceeds the upper limit, the shooting control unit 309 updates the target Ev value to a value decreased by 1 in S712. That is, the shooting control unit 309 desensitizes by decreasing the ISO correction step number by one. That is, the shooting control unit 309 determines the ISO sensitivity one step lower as the second exposure setting.

[0062] In S713, the shooting control unit 309 judges whether or not the ISO sensitivity determined in S712 has reached the lower limit. If the shooting control unit 309 judges that the ISO sensitivity has reached the lower limit, the processing proceeds to S714. If the shooting control unit 309 judges that the ISO sensitivity has not reached the lower limit, the processing returns to S703, and the shooting control unit 309 re-derives (provisionally determines) a Tv value to be the second exposure setting for the updated target Ev value.

[0063] In S714, the shooting control unit 309 updates the target Av value to a value increased by 1. That is, the shooting control unit 309 updates the target Av value to an aperture value that is narrowed down by one stop.

[0064] In S715, the photographing control unit 309 judges whether the target Av value updated in S714 has reached the upper limit of the Av value. Here, the upper limit of the Av value is set to be equal to or smaller than the airy disk diameter in the exposure setting of the visible light image. That is, the upper limit of the Av value is the photographing condition in which the aperture 205 is narrowed to the maximum within a range in which the effect of diffraction is considered to be equivalent to that in the photographing of the visible light image in the photographing of the near-infrared light image. If the photographing control unit 309 judges that the updated target Av value has reached the upper limit, the photographing control unit 309 moves the process to S716. If the photographing control unit 309 judges that the updated target Av value has not reached the upper limit, the photographing control unit 309 returns the process to S703 and re-derives (provisionally determines) the Tv value to be the second exposure setting based on the updated target Av value and the updated target Ev value.

[0065] In S716, the shooting control unit 309 determines the upper limit of the Av value setting as the second Av value.

[0066] In S717, the shooting control unit 309 determines the determined second Av value, and the Tv value and ISO sensitivity determined at the time of reaching this step as the second exposure setting.

[0067] In S718, the shooting control unit 309 determines the number of times to shoot in focus bracketing shooting of near-infrared images and the focus position to be set for each time based on the determined second Av value, and includes that information in the second exposure setting to complete this determination process. Here, the information on the focus position to be set for each time may be information on the amount of movement of the focusing lens 203.

[0068] When each parameter of the second exposure setting is determined in this manner in the determination process, the CPU 301 advances the process to S603 of the shooting process.

[0069] In S603, under the control of the CPU 301, the shooting control unit 309 supplies each parameter of the first exposure setting determined in S601 to the imaging control unit 310 and the optical system control unit 311, and causes them to shoot a visible light image based on the first exposure setting. That is, the aperture driving unit 204 drives the aperture 205 to an aperture amount corresponding to the first Av value determined as the first exposure setting, and the imaging control unit 310 exposes the pixel array 3041 for a time period according to the Tv value determined as the first exposure setting. The imaging unit 304 outputs image data (visible light image) to which a gain corresponding to the ISO sensitivity determined as the first exposure setting has been applied under the control of the imaging control unit 310.

[0070] In S604, under the control of the CPU 301, the shooting control unit 309 supplies each parameter of the second exposure setting determined in S602 to the imaging control unit 310 and the optical system control unit 311, and causes focus bracket shooting of a near-infrared light image to be performed based on the second exposure setting. That is, the lens driving unit 202 drives the focusing lens 203 to a position corresponding to the in-focus position of each shooting, and the aperture driving unit 204 drives the aperture 205 to an aperture amount corresponding to the second Av value determined as the second exposure setting. The imaging control unit 310 also exposes the pixel array 3041 for a time period corresponding to the Tv value determined as the second exposure setting. The imaging unit 304 outputs image data (near-infrared light image) to which a gain corresponding to the ISO sensitivity determined as the second exposure setting is applied under the control of the imaging control unit 310. Since this is focus bracket shooting, the lens driving unit 202 moves the focusing lens 203 to different positions in sequence for each shooting, and the imaging unit 304 outputs a near-infrared light image for each shooting. When all the near-infrared light images have been captured, the CPU 301 completes the main capturing process.

[0071] As described above, the imaging device 100 of this embodiment can determine exposure settings for acquiring visible light images and near-infrared light images with small deviations in resolution. More specifically, at least when capturing a near-infrared light image, an aperture value that is more open than the aperture value used when capturing a visible light image is set, so that a visible light image and a near-infrared light image can be output with reduced deviations in the effects of diffraction due to differences in wavelength between visible light and near-infrared light.

[0072] Although the imaging device 100 of this embodiment has been described as a lens-interchangeable type capable of capturing visible light images and near-infrared light images by mounting the lens barrel 200 on the main body 300, the implementation of the present invention is not limited to this. The imaging optical system may be provided integrally with the main body 300. Therefore, the positional relationship between the focusing lens 203 and the aperture 205 in the lens barrel 200 shown in FIG. 1 and the various optical elements provided on the optical axis may be configured in any manner.

[0073] In the present embodiment, the representative wavelength of the visible light region is set based on the reference wavelength of the optics and the optical device, and the representative wavelength of the near-infrared light is set based on the spectral sensitivity characteristics of the image sensor, but the present invention is not limited to this. The representative wavelength of each wavelength region may be configured to be set by the user, for example.

[0074] [Variation 1] In the above embodiment, the focus bracketing shooting is performed for near-infrared light, but the present invention is not limited to this. The present invention does not exclude a mode in which only a near-infrared light image obtained by single-shot shooting is output for near-infrared light as in the case of visible light. For example, when the shooting control unit 309 determines that the depth distribution of the subject falls within the depth of field related to the second aperture value, the second exposure setting may be configured to perform single-shot shooting for near-infrared light instead of focus bracketing shooting. In this case, the second exposure setting may be configured to include information indicating that one normal shooting (single-shot shooting) is performed instead of information on multiple shootings, or may be configured to include only information on one type of focus position instead of information on multiple types of focus positions.

[0075] [Variation 2] In the shooting process of the embodiment described above, only a single shot of visible light is shot, but the present invention is not limited to this. The present invention does not exclude a mode in which focus bracketing shooting is performed for visible light and multiple visible light images are output.

[0076] [Variation 3] In the above-mentioned embodiment and modified example, a mode in which a visible light image and a near-infrared light image can be captured using one imaging element by applying R, G, B, and IR filters to each pixel of the pixel array 3041 has been described, but the present invention is not limited to this embodiment. For example, as shown in FIG. 8, an imaging unit 801 for capturing visible light and an imaging unit 802 for capturing near-infrared light are provided in the main body 300, and the former is configured to capture a visible light image and the latter is configured to capture a near-infrared light image. Such a configuration of the imaging units 801 and 802 can be realized by, for example, adopting an imaging element to which a Bayer array color filter is applied in the former and an imaging element capable of capturing monochrome images without a color filter in the latter. That is, the former imaging element shows the spectral sensitivity characteristic shown in FIG. 9(a), and the latter imaging element shows the spectral sensitivity characteristic shown in FIG. 9(b).

[0077] In addition, when a CMOS sensor is used as the imaging element, the imaging element has sensitivity in the near-infrared wavelength range (780 nm to 1000 nm). Therefore, in a mode in which the same CMOS sensor is used for the imaging units 801 and 802, a similar configuration can be realized by applying an infrared cut filter to the former and a bandpass filter that passes near-infrared light to the latter. In other words, the imaging units 801 and 802 differ in whether the light components that each pixel of the pixel array 3041 can record are R, G, or B, or only IR.

[0078] In the illustrated example, light incident through the lens barrel 200 is dispersed using a half mirror 800 to form an image on each imaging unit, thereby enabling imaging. The imaging surface of the pixel array 3041 of the imaging unit 801 and the imaging surface of the pixel array 3041 of the imaging unit 802 are disposed at a conjugate position via the half mirror 800. In this case, the imaging control unit 309 supplies each parameter of the first exposure setting to the imaging control unit 310 and the optical system control unit 311, while supplying each parameter of the second exposure setting to the imaging control unit 803 and the optical system control unit 311. The imaging control unit 803 may have the same functional configuration as the imaging control unit 310, and controls the operation of the imaging unit 802 based on the input parameters of the second exposure setting (Tv value and ISO sensitivity).

[0079] [Embodiment 2] Incidentally, when the main body 300 is configured as shown in FIG. 8, it is also possible to capture a visible light image and a near-infrared light image without time division. However, the optical images formed on the image sensor of the image sensor of the image sensor of the image sensor 801 and the image sensor of the image sensor of the image sensor 802 via the half mirror 800 have different focal positions because the wavelengths of visible light and near-infrared light are different. For this reason, when both visible light and near-infrared light are photographed in a single shot, it is necessary to move the focus adjustment lens 203 to the corresponding focal positions, making it difficult to photograph two types of images at the same time. On the other hand, when both visible light and near-infrared light are focus-bracketed, a visible light image and a near-infrared light image can be photographed simultaneously in a photographing opportunity that employs the same focal position. That is, in a section where the range in which the focus adjustment lens 203 is driven for focus bracketing of visible light overlaps with the range in which the focus adjustment lens 203 is driven for focus bracketing of near-infrared light, two types of images can also be photographed simultaneously. That is, the efficiency of imaging can be improved by merging the range in which the focusing lens 203 is driven for focus bracket imaging and capturing visible light images and near-infrared light images at the same time. In other words, when performing focus bracket imaging of visible light and near-infrared light, the process of moving the focusing lens 203 can be shared, making it possible to efficiently obtain visible light images and near-infrared light images at some of the in-focus positions.

[0080] On the other hand, as described above, in the first exposure setting and the second exposure setting, the second aperture value included in the latter is set to the open side compared to the first aperture value included in the former. That is, in order to simultaneously capture the light beam incident through one imaging optical system by the imaging unit 801 and the imaging unit 802 as in the imaging device 100 of this embodiment, it is necessary to adapt not only the range in which the focus adjustment lens 203 is driven but also the aperture value to the shooting. Here, if the aperture value is set to the first aperture value, the resolution of the near-infrared light image may be lost due to the influence of diffraction, so the imaging device 100 of this embodiment performs focus bracket shooting of visible light and near-infrared light at the second aperture value. That is, since the visible light image is shot by setting the aperture value to the open second aperture value, the depth of field is narrower than when shooting with the first aperture value, and the focus position for shooting in focus bracket shooting needs to be set based on the second aperture value. Therefore, when performing focus bracketing shooting of both visible light and near-infrared light, the shooting control unit 309 determines the second aperture value based on the first aperture value, and then determines the focus position when performing focus bracketing shooting of visible light and near-infrared light based on the second aperture value.

[0081] 《Shooting Processing》 Hereinafter, a specific photographing process for outputting a visible light image and a near-infrared light image of a subject in the imaging device 100 of this embodiment will be described with reference to the flowchart in Fig. 10. The process corresponding to the flowchart can be realized by the CPU 301 reading out a corresponding processing program stored in, for example, the storage device 302, and executing the program by expanding it in the memory 303. This photographing process will be described as being started when it is detected that a photographing instruction for two types of light has been received via a user interface (not shown), for example.

[0082] It is assumed that photometry by the photometry unit 306 and distance measurement by the distance measurement unit 308 have been performed prior to the start of the actual photographing process. It is also assumed that the focus adjustment lens has been moved to a position where the subject is focused prior to the start of the actual photographing process. In the photographing process of this step, steps that perform the same processes as in the photographing process of embodiment 1 are given the same reference numbers and their explanations are omitted, and only steps that perform operations unique to this embodiment will be explained below.

[0083] When each parameter of the first exposure setting is determined in S601, the shooting control unit 309, under the control of the CPU 301, determines in S1001 whether or not to perform focus bracket shooting for visible light. The determination in this step can be made based on whether or not the distribution (depth) of the subject in the depth direction is outside the range of the depth of field associated with the first aperture value (does not fall within the depth of field). If the shooting control unit 309 determines to perform focus bracket shooting for visible light, it shifts the process to S1002, and if it determines not to perform focus bracket shooting, it shifts the process to S1003.

[0084] In S1002, the shooting control unit 309 stores information indicating that focus bracket shooting is to be performed in the memory 303 under the control of the CPU 301. Note that, at the start of this shooting process, it is assumed that the initial value is set such that focus bracket shooting is not performed (single shooting is performed) for visible light and near infrared light.

[0085] In S1003, under the control of the CPU 301, the shooting control unit 309 executes a determination process for determining each parameter of the second exposure setting based on the input second Lv value and the first Av value determined in S601.

[0086] Decision Processing Here, the decision processing executed in this step will be described in detail with reference to the flowchart in Fig. 11. In the description of this decision processing, steps that perform the same processing as the decision processing in the first embodiment will be given the same reference numbers and their description will be omitted, and only steps that perform operations unique to this embodiment will be described below.

[0087] When the parameters of the second exposure setting are determined in S717, the shooting control unit 309 determines in S1101 whether or not to perform focus bracket shooting for near-infrared light. The determination in this step can be made based on whether or not the distribution (depth) of the subject in the depth direction is outside the range of the depth of field related to the second aperture value (does not fall within the depth of field). Note that when it is determined to perform focus bracket shooting for visible light, it is also possible to determine to perform focus bracket shooting for near-infrared light as well, since the depth of the same subject is photographed for near-infrared light with the second aperture value that is more open. If the shooting control unit 309 determines to perform focus bracket shooting for near-infrared light, it shifts the process to S1102, and if it determines not to perform focus bracket shooting, it shifts the process to S1105.

[0088] In S1102, the shooting control unit 309 determines the number of times to shoot in focus bracketing of near-infrared images and the focus position to be set for each shooting, and includes that information in the second exposure setting.

[0089] In S1103, the shooting control unit 309 determines whether or not it has been decided to perform focus bracketing shooting for visible light. This determination in this step can be made based on whether or not information to that effect has been stored in the memory 303. If it has been decided to perform focus bracketing shooting for visible light, the shooting control unit 309 moves the process to S1104. If it has not been decided, that is, if it has been decided to perform single-shot shooting for visible light, the shooting control unit 309 ends this determination process.

[0090] In S1104, the shooting control unit 309 determines the number of times to shoot and the focus position to be set for each shooting to simultaneously perform focus bracket shooting of visible light images and near-infrared light images, and completes this determination process. That is, the shooting control unit 309 merges the ranges over which the focus adjustment lens 203 is moved in focus bracket shooting of visible light images and focus bracket shooting of near-infrared light images, and ultimately specifies the movement range of the focus adjustment lens 203 that covers both. The shooting control unit 309 then determines the focus position for each shooting based on the movement range.

[0091] As described above, visible light and near-infrared light have different wavelengths and therefore different focus ranges. In other words, in order to complete focus bracketing imaging of visible light and near-infrared light within the period in which the focus adjustment lens 203 moves once through that range, the imaging control unit 309 needs to set the lens movement range so that it covers both of the movement ranges that should be moved by each. This is also an adjustment to absorb the difference in the focus position of visible light and near-infrared light that occurs due to chromatic aberration.

[0092] Here, in order to simultaneously perform focus bracketing of a visible light image and a near-infrared light image during one movement of the focusing lens 203, the image capturing device 100 of this embodiment changes the first aperture value included in the first exposure setting to the second aperture value determined for near-infrared light. Therefore, in the image capturing device 100 of this embodiment, the number of times of shooting in focus bracketing of visible light images and information on the focus position are determined in this determination process. By determining the number of times of shooting in focus bracketing of visible light images based on the second aperture value, the number of times can be increased compared to when it is determined using the first aperture value, but the control of the lens driving unit 202 and each image capturing unit can be simplified.

[0093] The shooting control unit 309 updates the first aperture value to the value of the second aperture value, and determines the number of shots and information on the focus position in focus bracketing of visible light images based on the depth of field related to the updated aperture value. At this time, for the focus range in which shooting has already been determined by moving the focus adjustment lens 203 in focus bracketing shooting of near-infrared light, the information on the focus position included in the second exposure setting is reused.

[0094] On the other hand, for a focus position that is not included in the focus range determined for near-infrared light, it can be determined based on, for example, the difference in the amount of movement of the focusing lens 203 between consecutive focus positions determined for focus bracketing of near-infrared light images. The shooting control unit 309 determines the lens position for the section in which the focusing lens 203 is moved in increments of the difference in the amount of movement for the focus range in which focus is achieved only in focus bracketing of visible light, and determines the focus position for each shooting accordingly. At this time, the shooting control unit 309 can also specify the number of times of shooting to be performed for the focus range related only to visible light. Therefore, the shooting control unit 309 can derive the number of times of shooting when performing the entire focus bracketing of visible light images and near-infrared light images by adding up the number of times of shooting to the number of times of shooting determined for near-infrared light. The number of times of shooting is the number of times the focusing lens 203 is moved to different focus positions in the entire focus bracketing of visible light images and near-infrared light images. The photographing control unit 309 outputs information on the number of photographs taken and the in-focus position for each photographing operation obtained in this manner to the imaging control unit 310 and the optical system control unit 311 .

[0095] In the imaging device 100 of this embodiment, it is not necessary to capture both the visible light image and the near-infrared light image at each focus position during the period in which the focusing lens 203 moves once. That is, the focus bracketing of the visible light image is performed during the period in which the focusing lens 203 is at a position corresponding to the focus position determined for visible light. Also, the focus bracketing of the near-infrared light image is performed during the period in which the focusing lens 203 is at a position corresponding to the focus position determined for near-infrared light. Therefore, both the visible light image and the near-infrared light image are captured only at a focus position determined commonly for both. Information regarding which of the imaging unit 801 and the imaging unit 802 will capture at which focus position is included in, for example, the first exposure setting and the second exposure setting.

[0096] On the other hand, if it is determined in S1101 that focus bracketing shooting will not be performed for near-infrared light, the shooting control unit 309 completes this decision process in S1105 by including information indicating that a near-infrared light image will be captured in a single shot in the second exposure setting.

[0097] When the determination process is completed in this manner, the shooting control unit 309, under the control of the CPU 301, determines a combination of shooting modes of the visible light image and the near-infrared light image in S1004 of the shooting process. If the shooting control unit 309 determines to perform single-shot shooting of both the visible light image and the near-infrared light image, it transfers the process to S1005. If the shooting control unit 309 determines to perform single-shot shooting of the visible light image and to perform focus bracket shooting of the near-infrared light image, it transfers the process to S1007. If the shooting control unit 309 determines to perform focus bracket shooting of both the visible light image and the near-infrared light image, it transfers the process to S1009.

[0098] In S1005, under the control of the CPU 301, the shooting control unit 309 supplies each parameter of the first exposure setting determined in S601 to the imaging control unit 310 and the optical system control unit 311, and causes single-shot shooting of a visible light image to be performed based on the first exposure setting. In response to this, the imaging unit 801 outputs image data of the visible light image under the control of the imaging control unit 310.

[0099] In S1006, under the control of the CPU 301, the shooting control unit 309 supplies each parameter of the second exposure setting determined in S1003 to the imaging control unit 310 and the optical system control unit 311, and causes single shooting of a near-infrared light image to be performed based on the second exposure setting. In response to this, the imaging unit 802 outputs image data of the near-infrared light image based on the control of the imaging control unit 310. When the near-infrared light image is output, the CPU 301 completes the main shooting process.

[0100] Furthermore, when single-shot shooting of a visible light image is performed and focus bracketing shooting of a near-infrared light image is performed, the shooting control unit 309, under the control of the CPU 301, performs control to perform single-shot shooting of a visible light image in S1007. That is, the shooting control unit 309 supplies each parameter of the first exposure setting determined in S601 to the imaging control unit 310 and the optical system control unit 311, and causes single-shot shooting of a visible light image to be performed based on the first exposure setting. In response to this, the imaging unit 801 outputs image data (visible light image) to which a gain corresponding to the ISO sensitivity determined as the first exposure setting is applied under the control of the imaging control unit 310.

[0101] In S1008, under the control of the CPU 301, the shooting control unit 309 supplies each parameter of the second exposure setting determined in S1003 to the imaging control unit 310 and the optical system control unit 311, and causes focus bracket shooting of a near-infrared light image to be performed based on the second exposure setting. In response to this, the imaging unit 802 outputs image data (near-infrared light image) to which a gain corresponding to the ISO sensitivity determined as the second exposure setting is applied under the control of the imaging control unit 310. Since this is focus bracket shooting, the lens driving unit 202 moves the focusing lens 203 sequentially to different positions for each shooting, and the imaging unit 802 outputs a near-infrared light image for each shooting. When the near-infrared light images for all shootings have been output, the CPU 301 completes the main shooting process.

[0102] On the other hand, when focus bracket shooting is performed for both visible light images and near-infrared light images, the shooting control unit 309 performs control to simultaneously perform focus bracket shooting of visible light images and near-infrared light images in S1009 under the control of the CPU 301. That is, the shooting control unit 309 supplies each parameter of the first exposure setting and the second exposure setting determined in S601 and S1003 to the imaging control unit 310 and the optical system control unit 311, and causes focus bracket shooting of the visible light image and the near-infrared light image to be performed based on these parameters.

[0103] The lens driving unit 202 drives the focusing lens 203 to a position corresponding to the in-focus position for each shooting, and the aperture driving unit 204 drives the aperture 205 to an aperture amount corresponding to a second Av value determined commonly for visible light and near-infrared light. The imaging control unit 310 also exposes the pixel array 3041 of the imaging unit 801 for a time period corresponding to the Tv value determined as the first exposure setting, and exposes the pixel array 3041 of the imaging unit 802 for a time period corresponding to the Tv value determined as the second exposure setting. The imaging unit 801 outputs image data of a visible light image to which a gain corresponding to the ISO sensitivity determined as the first exposure setting has been applied under the control of the imaging control unit 310. The imaging unit 802 also outputs image data of a near-infrared light image to which a gain corresponding to the ISO sensitivity determined as the second exposure setting has been applied under the control of the imaging control unit 310.

[0104] Since this is focus bracketing photography, the lens driving unit 202 moves the focusing lens 203 to different positions in sequence for each photography round, and the image capturing units 801 and 802 output image data for the photography rounds designated for photography. The CPU 301 completes this photography process when at least one of visible light images and near-infrared light images has been output for all photography rounds.

[0105] In this way, the imaging device 100 of this embodiment can improve the efficiency of control in a mode in which a visible light image and a near-infrared light image are acquired by focus bracketing imaging.

[0106] In this embodiment, in the case where both visible light images and near-infrared light images are focus-bracketed, the imaging unit with the corresponding focus position is used for each shooting, but the present invention is not limited to this. For example, both visible light images and near-infrared light images may be captured in all shootings, and only necessary images (images included in the focus range) may be selected from these images at the time of output. Alternatively, for example, when generating a composite image, only necessary images may be selected as the composite target, thereby excluding images that do not favorably show the state of the subject.

[0107] In this embodiment, the number of times of shooting and the in-focus position for each shooting are derived in the focus bracketing shooting of both visible light images and near-infrared light images in S1104 of the determination process, but the embodiment of the present invention is not limited to this. Since the number of times of shooting and the change in the in-focus position accompanying the change in the aperture value are determined according to, for example, the relationship between the representative wavelength in the visible light region and the representative wavelength in the near-infrared light region, they may be measured / derived in advance for each of a plurality of types of representative subject distances and stored in the storage device 302 for use.

[0108] [Variation 4] In the above-mentioned embodiment and modified example, the imaging device 100 has been described as being placed in an environment in which the subject can be photographed using visible light and near-infrared light. However, for example, when photographing indoors using fluorescent lighting, the irradiated light may not include light in the near-infrared region. In addition, when an obstruction is placed between the subject and the light source of visible light, the luminance of the subject may not be stable, and the obtained visible light image may not be effectively used in, for example, machine vision. For this reason, the imaging device 100 may be configured to include various light sources that irradiate the subject with corresponding light in synchronization with photographing. For example, a white LED may be used as the light source of visible light. For example, a near-infrared emitting LED (hereinafter referred to as a near-infrared LED) may be used as the light source of near-infrared light.

[0109] When such light emitting elements are used, the representative wavelength of each wavelength range in the captured image may be set according to the light emitting element used. The white LED exhibits, for example, the spectral characteristics shown in FIG. 12(a). As shown in the figure, the light of the white LED has a sharp peak in the blue region, but the representative wavelength of the range of 400 nm to 780 nm may be determined to be around 560 nm in the center. The near-infrared LED exhibits, for example, the spectral characteristics shown in FIG. 12(b). As shown in the figure, the representative wavelength of the light of the near-infrared LED may be determined to be around 830 nm, which is the center of emission.

[0110] In the imaging device 100 having such a light source, for example, when the photometry unit 306 measures light, each light-emitting element is controlled to emit light. The control of each light-emitting element may be performed by, for example, a light emission control unit (not shown) that operates in response to an instruction from the shooting control unit 309 or the CPU 301. The light emission control unit causes the white LED and the near-infrared LED to emit light as pre-emission during photometry, and the imaging unit 304 captures an object illuminated by these LEDs and outputs an image used for photometry by the photometry unit 306. At this time, the white LED and the near-infrared LED can be emitted simultaneously because their spectral characteristics do not overlap as shown in Figs. 12(a) and 12(b). The light emission control unit may also control the emission of each LED so that the luminance of an arbitrary area in the image becomes a predetermined light amount. Here, the predetermined light amount can be defined by regarding the obtained Lv value as an Ev value, for example, so that it corresponds to an arbitrary position in the table of Fig. 5 at the open F value. Note that by controlling the light amount at this timing, it is possible to easily determine each parameter of the exposure setting in the determination process, etc.

[0111] Furthermore, when a visible light image and a near-infrared light image are captured in the imaging process, the light emission control unit controls the corresponding LEDs to emit light.

[0112] Although the imaging device 100 of this modified example has been described as having a light source that emits light continuously, such as an LED, it may be configured to have a light source that emits light instantaneously, such as a strobe light. If the light source is a strobe light, the guide number is defined, and the aperture value (F value) can uniquely define the relationship between the amount of light emitted, the subject distance, and the ISO sensitivity. Therefore, if the shutter speed is made slower than the strobe synchronization speed, photometry processing before the shooting process is not necessary (however, the options for the Tv value are narrowed).

[0113] In addition, in this modification, a configuration is described in which the imaging device 100 is provided with light sources that irradiate light in two wavelength ranges for image capture, but the embodiment of the present invention is not limited to this. At least one of the light sources may be provided outside the imaging device 100 and configured to irradiate the subject in response to a control signal output from the imaging device 100. The external light source may also be a light source that can irradiate light in multiple wavelength ranges. In this case, the light emission control unit supplies information on the wavelength range and representative wavelength to be irradiated by the external light source. The information on the wavelength range and representative wavelength to be irradiated by the external light source may be configured to be set by the user via a GUI in the main body unit 300, for example.

[0114] [Variation 5] In the above-mentioned embodiment and modified example, the imaging device 100 is described as being configured to be capable of capturing a visible light image and a near-infrared light image through one imaging optical system, but the implementation of the present invention is not limited to this. In the present invention, as long as a more open aperture value is determined for near-infrared light than that for capturing a visible light image in order to reduce the discrepancy in the influence of diffraction between visible light and near-infrared light, the visible light image and the near-infrared light image do not need to be captured by the same imaging optical system. For example, the present invention can be implemented even in a mode in which different imaging optical systems form optical images for an imaging element that captures a visible light image and an imaging element that captures a near-infrared light image, respectively.

[0115] [Variation 6] In the above-mentioned embodiment and modified example, the aspect of determining the exposure setting when photographing light in the visible light region and light in the near-infrared light region has been described. However, the implementation of the present invention is not limited to this, and light in other wavelength regions may be used, for example. That is, the object of the present invention is to reduce the deviation of the influence of diffraction between images obtained by photographing each of light in two types of wavelength regions, and the present invention is applicable if there is a difference in the wavelength region, such as a difference in the representative wavelength. Therefore, the present invention only needs to determine the second aperture value, which is more open than the first aperture value determined for the former photographing, as the exposure setting for the latter photographing when photographing a subject with light in a first wavelength region and light in a second wavelength region that includes a wavelength longer than the first wavelength region.

[0116] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0117] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0118] [Summary of the embodiment and modifications] The disclosure of this specification includes the following information processing device, imaging device, control method, and program. (Item 1) An information processing device that determines exposure settings of an imaging device that captures an image of a subject using light of two wavelength ranges, a first acquisition means for acquiring photometric results of the subject for each of light in a first wavelength range and light in a second wavelength range that includes a wavelength longer than that of the first wavelength range; a first determination means for determining a first exposure setting for photographing the subject with light in the first wavelength range based on a photometric result of the light in the first wavelength range; a second determination means for determining a second exposure setting for photographing the subject with light in the second wavelength range based on a photometric result of the light in the second wavelength range and the first exposure setting; having Exposure settings include aperture value, The second determination means determines, for the second exposure setting, a second aperture value that is more open than the first aperture value determined by the first determination means. 23. An information processing apparatus comprising: (Item 2) 2. The information processing device according to item 1, wherein the second exposure setting includes information on a focus position used for each of a plurality of photographing operations. (Item 3) Further comprising a second acquisition means for acquiring a distribution of the subject in the depth direction, The second determination means determines a focus position for each of the plurality of photographing operations based on a depth of field based on the second aperture value and a distribution of the subject in the depth direction. 3. The information processing device according to item 2. (Item 4) The information processing device described in item 3, characterized in that the second determination means determines the number of times to photograph the subject using light in the second wavelength range based on a depth of field based on the second aperture value and a depth-wise distribution of the subject. (Item 5) The information processing device described in item 3 or 4, characterized in that the second determination means includes, in the second exposure setting, information on a focus position used in one shooting instead of information on the focus positions in the multiple shootings when the subject falls within a depth of field based on the second aperture value. (Item 6) The information processing device described in any one of items 3 to 5, characterized in that the second determination means includes information in the second exposure setting indicating that a single shooting will be performed instead of the multiple shootings when the subject falls within a depth of field based on the second aperture value. (Item 7) The method further includes an evaluation means for evaluating the influence of diffraction occurring during imaging, The second determination means determines the second aperture value so that an evaluation value of an influence of diffraction when photographing light in the second wavelength range does not exceed an evaluation value of an influence of diffraction when photographing light in the first wavelength range at the first aperture value. 7. The information processing device according to any one of items 1 to 6, (Item 8) 8. The information processing device according to item 7, wherein the evaluation value of the effect of diffraction is an Airy disk diameter on an imaging surface of an imaging element that captures light in each wavelength range. (Item 9) 9. The information processing device according to item 7 or 8, wherein the first wavelength range is a visible light range. (Item 10) 10. The information processing device according to item 9, wherein the second wavelength range is a near-infrared light range. (Item 11) 11. The information processing device according to item 9 or 10, wherein the second wavelength range is from 780 nm to 1000 nm. (Item 12) Item 10 or 11, wherein the evaluation means derives an evaluation value of an effect of diffraction when photographing light in the first wavelength range based on a representative wavelength of the first wavelength range, and derives an evaluation value of an effect of diffraction when photographing light in the second wavelength range based on a representative wavelength of the second wavelength range. (Item 13) 13. The information processing device according to item 12, wherein the representative wavelength of the first wavelength range is a reference wavelength of an optical device. (Item 14) 14. The information processing device according to item 12 or 13, wherein the representative wavelength of the first wavelength range is the wavelength of e-line (546.07 nm) or d-line (587.56 nm) defined in ISO7944:1998. (Item 15) The information processing device described in any one of items 12 to 14, characterized in that the representative wavelength of the second wavelength range is a wavelength included in a peak band of the spectral sensitivity characteristics of an image sensor used to capture light in the second wavelength range. (Item 16) 16. The information processing device according to any one of items 1 to 15, wherein the second determination means determines the shutter speed and sensitivity of the second exposure setting based on the second aperture value so as to appropriately expose the subject. (Item 17) An information processing device according to any one of items 1 to 16, a first imaging means for imaging the subject using light in the first wavelength range; a second imaging means for imaging the subject using light in the second wavelength range; a control means for causing the first imaging means to capture an image of the subject based on the first exposure setting determined by the first determination means, and causing the second imaging means to capture an image of the subject based on the second exposure setting determined by the second determination means; An imaging device comprising: (Item 18) Item 18. The imaging device described in item 17, characterized in that the control means outputs an image of light in the first wavelength range obtained by imaging with the first imaging means and an image of light in the second wavelength range obtained by imaging with the second imaging means in association with each other. (Item 19) The method further includes a generating unit that generates a composite image by combining a plurality of images of light in the second wavelength range obtained by the plurality of images taken by the second imaging unit when the second imaging unit takes a plurality of images, 20. The imaging device according to item 18, wherein the control means outputs the image of the light in the first wavelength range and the composite image in association with each other. (Item 20) the first imaging means and the second imaging means are configured as a single imaging element in which light incident via an imaging optical system having a common aperture and a focus adjustment lens is imaged; The control means controls the first imaging means and the second imaging means to capture images in a time-division manner. 20. The imaging device according to any one of items 17 to 19, (Item 21) the first imaging means and the second imaging means are configured as different imaging elements in which light incident via an imaging optical system having a common aperture and a focusing lens is dispersed and imaged, The first determination means determines the first exposure setting by using the second aperture value instead of the first aperture value when the control means causes the first imaging means and the second imaging means to simultaneously perform imaging. 20. The imaging device according to any one of items 17 to 19, (Item 22) A method for controlling an information processing device that determines exposure settings of an imaging device that captures an image of a subject using light of two wavelength ranges, comprising: a first acquisition step of acquiring photometric results of the subject for each of light in a first wavelength range and light in a second wavelength range that includes a wavelength longer than the first wavelength range; a first determination step of determining a first exposure setting for photographing the subject with light in the first wavelength range based on a photometric result of the light in the first wavelength range; a second determination step of determining a second exposure setting for photographing the subject with light in the second wavelength range based on a photometric result of the light in the second wavelength range and the first exposure setting; having Exposure settings include aperture value, In the second determination step, a second aperture value that is more open than the first aperture value determined in the first determination step is determined for the second exposure setting. A control method comprising: (Item 23) A program for causing a computer to function as each of the means of the information processing device according to any one of items 1 to 16. [Explanation of symbols]

[0119] 100: imaging device, 203: focusing lens, 205: aperture, 301: CPU, 304: imaging section, 306: photometry section, 308: distance measurement section, 309: photography control section, 310: imaging control section, 311: optical system control section

Claims

1. An information processing device that determines exposure settings of an imaging device that captures an image of a subject using light of two wavelength ranges, a first acquisition means for acquiring photometric results of the subject for each of light in a first wavelength range and light in a second wavelength range that includes a wavelength longer than the first wavelength range; a first determination means for determining a first exposure setting for photographing the subject with light in the first wavelength range based on a photometric result of the light in the first wavelength range; a second determination means for determining a second exposure setting for photographing the subject with light in the second wavelength range based on a photometric result of the light in the second wavelength range and the first exposure setting; having Exposure settings include aperture value, The second determination means determines a second aperture value for the second exposure setting that is more open than the first aperture value determined by the first determination means.

23. An information processing apparatus comprising:

2. 2 . The information processing apparatus according to claim 1 , wherein the second exposure setting includes information on a focus position for each of a plurality of photographing operations.

3. The method further includes a second acquisition means for acquiring a distribution of the subject in a depth direction, The second determination means determines a focus position for each of the plurality of photographing operations based on a depth of field based on the second aperture value and a distribution of the subject in the depth direction.

3. The information processing apparatus according to claim 2.

4. 4. The information processing device according to claim 3, wherein the second determination means determines the number of times to photograph the subject using light in the second wavelength range based on a depth of field based on the second aperture value and a depth-wise distribution of the subject.

5. The information processing device according to claim 3, characterized in that, when the subject falls within a depth of field based on the second aperture value, the second determination means includes information on the focus position used in one shooting in the second exposure setting instead of information on the focus position in the multiple shootings.

6. 4. The information processing device according to claim 3, characterized in that the second determination means includes, in the second exposure setting, information indicating that a single shooting is performed instead of the multiple shootings when the subject falls within a depth of field based on the second aperture value.

7. The method further includes an evaluation means for evaluating the influence of diffraction occurring during imaging, The second determination means determines the second aperture value so that an evaluation value of an influence of diffraction when photographing light in the second wavelength range does not exceed an evaluation value of an influence of diffraction when photographing light in the first wavelength range at the first aperture value.

2. The information processing apparatus according to claim 1,

8. 8. The information processing apparatus according to claim 7, wherein the evaluation value of the effect of diffraction is an Airy disk diameter on an imaging surface of an imaging element that captures light in each wavelength range.

9. 8. The information processing apparatus according to claim 7, wherein the first wavelength range is a visible light range.

10. 10. The information processing apparatus according to claim 9, wherein the second wavelength range is a near-infrared light range.

11. 10. The information processing apparatus according to claim 9, wherein the second wavelength range is from 780 nm to 1000 nm.

12. The information processing device according to claim 10, characterized in that the evaluation means derives an evaluation value of the effect of diffraction when photographing light in the first wavelength range based on a representative wavelength of the first wavelength range, and derives an evaluation value of the effect of diffraction when photographing light in the second wavelength range based on a representative wavelength of the second wavelength range.

13. 13. The information processing apparatus according to claim 12, wherein the representative wavelength of the first wavelength range is a reference wavelength of an optical device.

14. 13. The information processing apparatus according to claim 12, wherein the representative wavelength of the first wavelength range is a wavelength of e-line (546.07 nm) or d-line (587.56 nm) defined in ISO 7944:1998.

15. 13 . The information processing apparatus according to claim 12 , wherein the representative wavelength of the second wavelength range is a wavelength included in a peak band of the spectral sensitivity characteristics of an image pickup element used for capturing an image with light of the second wavelength range.

16. 2. The information processing apparatus according to claim 1, wherein the second determination means determines a shutter speed and a sensitivity of the second exposure setting based on the second aperture value so as to provide an appropriate exposure for the subject.

17. An information processing device according to any one of claims 1 to 16, a first imaging means for imaging the subject using light in the first wavelength range; a second imaging means for imaging the subject using light in the second wavelength range; a control means for causing the first imaging means to capture an image of the subject based on the first exposure setting determined by the first determination means, and causing the second imaging means to capture an image of the subject based on the second exposure setting determined by the second determination means; An imaging device comprising:

18. 18. The imaging device according to claim 17, wherein the control means outputs an image of light in the first wavelength range obtained by imaging with the first imaging means and an image of light in the second wavelength range obtained by imaging with the second imaging means in association with each other.

19. The method further includes a generating unit that generates a composite image by combining a plurality of images of light in the second wavelength range obtained by the plurality of images taken by the second imaging unit when the second imaging unit takes a plurality of images, 20. The imaging device according to claim 18, wherein the control means outputs the image of the light in the first wavelength range and the composite image in association with each other.

20. the first imaging means and the second imaging means are configured as a single imaging element in which light incident via an imaging optical system having a common aperture and a focus adjustment lens is imaged; The control means controls the first imaging means and the second imaging means to perform imaging in a time-division manner.

20. The imaging device according to claim 17,

21. the first imaging means and the second imaging means are configured as different imaging elements in which light incident via an imaging optical system having a common aperture and a focusing lens is dispersed and imaged, The first determination means determines the first exposure setting by using the second aperture value instead of the first aperture value when the control means causes the first imaging means and the second imaging means to simultaneously perform imaging.

20. The imaging device according to claim 17,

22. A method for controlling an information processing device that determines exposure settings of an image capturing device that captures an image of a subject using light of two wavelength ranges, comprising: a first acquisition step of acquiring photometric results of the subject for each of light in a first wavelength range and light in a second wavelength range that includes a wavelength longer than the first wavelength range; a first determination step of determining a first exposure setting for photographing the subject with light in the first wavelength range based on a photometric result of the light in the first wavelength range; a second determination step of determining a second exposure setting for photographing the subject with light in the second wavelength range based on a photometric result of the light in the second wavelength range and the first exposure setting; having Exposure settings include aperture value, In the second determination step, a second aperture value that is more open than the first aperture value determined in the first determination step is determined for the second exposure setting. A control method comprising:

23. A program for causing a computer to function as each of the means of the information processing apparatus according to any one of claims 1 to 16.

Images