Image processing apparatus, image processing method, program, and storage medium

Abstract

To reduce variations in the colors of an image for every characteristic of a photographing lens in infrared photography.SOLUTION: An image processing apparatus comprises: a first acquisition unit that acquires information on an imaging optical system; a second acquisition unit that acquires an image signal from an image pick-up device; and an image processing unit that creates an image in a first mode for creating an achromatic color image or a single color image on the basis of the image signal, or a second mode for creating an image including a plurality of chromatic colors on the basis of the image signal. The image processing unit determines one of the first mode and the second mode to create an image on the basis of the information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, a program, and a storage medium. [Background technology]

[0002] Conventionally, imaging devices are known that perform white balance control with an imaging sensor capturing infrared light components and output color images. The color of a color image in a normal shooting mode is designed to capture only visible light components without capturing infrared light components. On the other hand, there are imaging devices equipped with a shooting mode in which, for shooting in a low-illumination environment, a shooting lens is specially processed to capture more infrared light components into the imaging sensor and improve sensitivity. However, as described above, since the color of the imaging device is designed not to capture infrared light components, the color of the color image is distorted from the original color of the subject by capturing infrared light components. Specifically, the color of the captured image becomes reddish compared to a state in which infrared light components are not captured. In addition, due to the special processing applied to the shooting lens, the characteristics for the wavelength of the non-visible light region including infrared light components differ for each shooting lens, and the color of the generated captured image changes for each shooting lens. In other words, during infrared shooting, the color of the output color image varies depending on the characteristics of the shooting lens used.

[0003] In order to deal with the color variation in captured images for each photographing lens, Patent Document 1 discloses a technique for calculating a white balance gain for a different lens unit based on already calculated color temperature information and information on a different photographing lens connected. According to Patent Document 1, the white balance gain is recalculated based on the difference in color temperature information calculated for each photographing lens, thereby absorbing the difference between the photographing lenses, thereby reducing the color variation during normal photographing.

[0004] Patent Document 2 discloses a technique for performing white balance adjustment based on optical characteristics by setting a white balance fine adjustment coefficient in response to changes in the optical characteristics of a photographing lens. According to Patent Document 2, the color cast phenomenon caused by white balance adjustment errors resulting from changes in the pupil position or aperture value of the photographing lens is suppressed, thereby reducing color variation.

[0005] In recent years, specially processed lenses have become capable of transmitting more infrared light components, and imaging elements are known that can obtain an image that is a mixture of visible light components and infrared light components when the infrared cut filter is removed.

[0006] However, the conventional techniques disclosed in the above-mentioned patent documents do not take into consideration color variations that occur when infrared photography is performed using a lens that has characteristics for the non-visible light region that includes infrared light components, and therefore there is a risk that the white balance performance of the captured image generated during infrared photography will vary depending on the lens used.

[0007] If one attempts to eliminate the variation in white balance performance during infrared photography, it will adversely affect the white balance performance in normal photography mode, in which an image consisting of only visible light components is captured when an infrared cut filter is inserted. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Patent Publication 2019-205003 [Patent Document 2] Patent Publication No. 2000-244930 Summary of the Invention [Problem to be solved by the invention]

[0009] Therefore, the problem to be solved by the present invention is to reduce the variation in color of captured images generated for each characteristic of the photographing lens during infrared photographing. [Means for solving the problem]

[0010] In order to solve the above problem, an image processing device according to one embodiment of the present invention includes a first acquisition unit that acquires information regarding an imaging optical system, a second acquisition unit that acquires an image signal from an imaging element, and an image processing unit that generates an image in a first mode that generates an achromatic or monochrome image based on the image signal or a second mode that generates an image including multiple chromatic colors based on the image signal, and is characterized in that the image processing unit determines in which mode, the first mode or the second mode, the image will be generated based on the information. Effect of the Invention

[0011] According to the present invention, it is possible to reduce the variation in image color due to the characteristics of the photographing lens during infrared photographing. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating a device configuration and a functional configuration of an imaging apparatus. [Diagram 2] 13 is a flowchart showing the flow of image processing. [Diagram 3] FIG. 2 is a diagram illustrating the transmittance characteristics of an imaging optical system. [Figure 4] FIG. 2 is a diagram illustrating the spectral sensitivity characteristics of an image sensor. [Diagram 5] FIG. 2 is a diagram illustrating a functional configuration of an image processing device. [Figure 6] 13 is a flowchart showing the flow of image processing. [Figure 7] FIG. 2 is a diagram illustrating the transmittance characteristics of an imaging optical system. [Figure 8] FIG. 11 is a diagram illustrating an output image in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0014] The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In addition, the embodiments may be configured by appropriately combining parts of each of the embodiments described below. In the following embodiments, the same components are described with the same reference numerals.

[0015] <First embodiment> FIG. 1(a) is a block diagram showing the device configuration of an imaging device according to this embodiment, and FIG. 1(b) is a block diagram showing the functional configuration of the imaging device according to this embodiment.

[0016] (Device configuration) The imaging device according to this embodiment has an imaging optical system 101, an optical filter 102, an imaging element (first acquisition unit) 103, a CPU 104, a RAM 105, and a ROM 106, and each unit is electrically connected via a bus 107. The imaging optical system 101 is a lens connected to the imaging device, and light incident on the imaging optical system 101 forms an image of a subject. The imaging device according to this embodiment acquires information about the imaging optical system 101 from the connected imaging optical system 101. Then, a mode for generating an appropriate image is determined based on the acquired information, and WB (white balance) control is performed according to the determined mode, thereby obtaining an appropriate captured image for the characteristics of the imaging optical system.

[0017] The imaging optical system 101 is a lens group consisting of one or more lenses. The imaging optical system 101 includes lenses that are coated to transmit infrared light components more, and lenses that are specially processed to suppress the effects of aberration. Furthermore, the imaging optical system 101 includes lenses that change optical characteristics, such as a teleconverter (extender), and lenses that can be designed to have different functions for each wavelength, such as a metalens.

[0018] The optical filter 102 is an infrared cut filter that attenuates wavelengths in the non-visible light region, a bandpass filter that allows only one specific wavelength region to pass, or a dual bandpass filter that allows multiple specific wavelength regions to pass. The optical filter 102 is arranged on the optical axis so that it can be inserted and removed, and is inserted and removed by a driving unit (not shown). For example, by inserting an infrared cut filter on the optical axis, it is possible to attenuate the infrared light component of the incident light and reduce the influence of the infrared light component on the image signal of the image sensor 103.

[0019] The imaging element 103 is a photoelectric conversion element that captures an image of a subject by the imaging optical system 101 and generates image data or an image signal consisting of a plurality of pixels. The image data or image signal includes information of a plurality of colors. The plurality of colors are, for example, red, green, and blue, and the image data is data that passes through color filters corresponding to each color provided in front of the imaging element and is converted into an electric signal by the imaging element. The color filters transmit not only visible light components corresponding to red, green, and blue, but also some infrared light components included in the invisible light region. The imaging element may be a complementary metal oxide semiconductor (CMOS), a charge-coupled device (CCD), a single photon avalanche diode (SPAD), or the like.

[0020] The CPU 104 is a central processing unit for controlling the imaging device in an integrated manner. The RAM 105 is a non-volatile memory, and provides a work area used by the CPU 104 when executing processing. The RAM 105 also functions as a frame memory and a buffer memory. The ROM 106 stores programs and image data for the CPU 104 to control the imaging device. In addition, the ROM 106 can store optical information (optical characteristics) of multiple types of imaging optical systems, for example.

[0021] (Functional configuration) The imaging device according to this embodiment has an optical filter control unit 111, a lens information acquisition unit (second acquisition unit) 112, a mode determination unit 113, a WB control unit 114, and an image processing unit 115. Each function is executed by software, and is realized by reading a program stored in a ROM (storage unit) 106 into a RAM 105 and executing the program by a CPU 104.

[0022] The optical filter control unit 111 controls insertion and removal of the optical filter 102. The insertion and removal of the optical filter 102 is controlled according to a setting specified by a user, or automatically according to the usage environment. For example, the insertion and removal of the infrared cut filter is automatically controlled according to the illuminance of the environment in which the imaging device is used, the magnitude of the ratio of infrared light components to visible light components, brightness information, etc. This makes it possible to switch between a normal shooting mode and an infrared shooting mode. The normal shooting mode is a mode in which an infrared cut filter is inserted on the optical axis and visible light is photographed, and the infrared shooting mode is a mode in which the infrared cut filter is removed from the optical axis and visible light + infrared light is photographed.

[0023] The lens information acquisition unit 112 acquires information about the connected imaging optical system from the imaging optical system 101. The information about the imaging optical system includes optical information (optical characteristics) of the imaging optical system. Here, the optical information is, for example, a transmittance indicating the percentage of light transmitted with respect to the wavelength of light, a reflectance indicating the percentage of light reflected, an absorptance indicating the percentage of light absorbed, lens magnification information, and focal length. The information about the imaging optical system may also be identification information for identifying which type of imaging optical system the connected imaging optical system is among multiple types of imaging optical systems. For example, it is ID information. The information may be provided by a user, not just when the information is acquired directly from the imaging optical system.

[0024] The ROM 106 can store optical information of a plurality of types of imaging optical systems, and each optical information can be associated with ID information and stored as a table. The lens information acquisition unit 112 may acquire optical information of the connected imaging optical system 101 from the ROM 106 by referring to the acquired ID information and the table stored in the ROM 106.

[0025] When the lens information acquisition unit 112 is unable to acquire information about the imaging optical system 101 from the imaging optical system 101 because operation is not guaranteed, for example, the lens information acquisition unit 112 acquires information notifying the fact that information cannot be acquired in addition to the connection status of the imaging optical system 101.

[0026] The mode determination unit 113 sets one of a first mode and a second mode based on information about the lens from the lens information acquisition unit 112. The first mode is an image processing mode for generating an achromatic or monochrome image based on image data and an image signal obtained from the imaging element 103. The second mode is an image processing mode for generating an image including multiple chromatic colors (hereinafter referred to as a color image) based on image data and an image signal obtained from the imaging element 103.

[0027] More specifically, in the first mode, a black-and-white image or a monochrome gradation image is generated using the luminance information of the image data, etc. In the mode for generating color images, an image may be generated by mixing multiple colors such as red, green, and blue, or may be generated from a combination of two colors such as red and blue.

[0028] The mode is determined when the infrared shooting mode is switched to. The mode is determined based on information about the imaging optical system 101 acquired by the lens information acquisition unit 112. For example, when information about the imaging optical system 101 can be correctly acquired and a shooting lens having the expected optical characteristics is connected, the first mode is selected, and a color image is generated in the infrared shooting mode. The expected optical characteristics are optical characteristics in a range in which there is no variation in color due to the performance of the WB. When the information about the shooting lens cannot be correctly acquired, there is a possibility that the color of the image generated during infrared shooting may vary, so the first mode is selected, and an achromatic or monochromatic image is generated during infrared shooting. In addition, when the imaging optical system 101 having optical characteristics exceeding the performance of the WB is connected, there is a possibility that the color of the image generated during infrared shooting may vary, so the first mode is selected to generate a monochromatic image.

[0029] Alternatively, it is also possible to switch the image generation mode according to the automatic filter control of the optical filter 102 described above. Specifically, in a low-illuminance environment, when a certain threshold is exceeded, the normal shooting mode is switched to an infrared shooting mode that generates a multi-color image. In addition, when the low-illuminance environment is one in which there is almost no visible light component or the ratio of the infrared light component exceeds a certain threshold, it is also possible to control the switching to the infrared shooting mode that generates a monochromatic image.

[0030] The WB control unit 114 performs WB control according to the image data and image signal from the image sensor 103 and the determination result of the mode determination unit 113, and outputs the image data and image signal to which the WB has been applied to the image processing unit 115. For example, when the image generation mode during infrared shooting is the first mode, the WB gain is set to a fixed value, and a black and white image or a monochromatic image set by the user is generated based on the input image data and image signal. In another example, when the mode during infrared shooting is a mode for generating a multi-color image, control is performed as normal WB in a normal shooting mode that does not include infrared light components, and control is performed as WB taking the infrared light components into consideration during infrared shooting.

[0031] The image processing unit 115 performs image processing on the image data and image signals from the WB control unit 114, and generates and outputs an output image. The image processing includes processing to reduce noise added to the image, processing to change the saturation and hue, processing to change the gamma curve, etc. In addition, these image processing operations are performed according to the shooting mode described above.

[0032] The mode determination unit 113, the WB control unit 114, and the image processing unit 115 can be implemented as one function, and the image processing unit 115 may have the functions of the mode determination unit 113 and the WB control unit 114. The functions described in Fig. 1(b) are functions executed by software only, and are functions executed by the CPU 104 (or an image processing engine, not shown), and therefore each function can be integrated or divided.

[0033] 2 is a flowchart showing the flow of infrared image processing executed by the imaging device of this embodiment. Each process in this flowchart is started by the CPU 104 executing a program stored in the ROM 106.

[0034] Fig. 3 shows an example of the transmittance of a possibly used photographing lens with respect to the wavelength of light. Fig. 4 shows an example of the spectral sensitivity characteristic of an image sensor provided in an image pickup device with respect to the wavelength of light. Hereinafter, an example of an image pickup device according to this embodiment and its effects will be described with reference to the flowchart in Fig. 2, the transmittance characteristic of the lens in Fig. 3, and the spectral sensitivity characteristic of the image sensor in Fig. 4.

[0035] First, in step S1, the lens information acquisition unit 112 acquires information indicating optical characteristics of the imaging optical system, such as an ID that can identify an individual from the imaging optical system 101, and transmittance or reflectance. For example, information on the imaging optical system having transmittance for the wavelength of light as shown in FIG. 3 is acquired. As an example, FIG. 3 shows the characteristics of a lens 1001 drawn by a solid line and a lens 1002 drawn by a dashed line. In FIG. 3, the vertical axis indicates the transmittance of the imaging optical system, and the horizontal axis indicates the wavelength of light passing through the imaging optical system in the range of 400 nm to 1000 nm. The lens 1001 has a high transmittance for the visible light region (around 400 nm to 750 nm) and a low transmittance for the non-visible light region (750 nm to 1000 nm). On the other hand, the lens 1002 has a low transmittance for the visible region and a high transmittance for the non-visible region.

[0036] Next, in step S2, it is determined whether to generate a color image or a monochrome image during infrared shooting based on information about the imaging optical system acquired by the lens information acquisition unit 112. For example, assume that the imaging element 103 has a spectral sensitivity characteristic as shown in FIG. 4. In FIG. 4, the vertical axis indicates the spectral sensitivity of the imaging element, and the horizontal axis indicates the wavelength of light received by the imaging element in the range of 400 nm to 1000 nm. The spectral curve 1101 drawn by the solid line is the spectral sensitivity of a red pixel having a peak near a wavelength of 630 nm. The spectral curve 1102 drawn by the dashed line is the spectral sensitivity of a green pixel having a peak near a wavelength of 530 nm. The spectral curve 1103 drawn by the dotted line is the spectral sensitivity of a green pixel having a peak near a wavelength of 450 nm.

[0037] Consider the case where the lens 1001 in Fig. 3 is connected to an image sensor having the spectral sensitivity in Fig. 4. In this case, the transmittance of the imaging optical system in the non-visible light region is not so high, and even if a color image is generated during infrared shooting, the colors are not significantly distorted, making it possible to output color during infrared shooting.

[0038] On the other hand, consider the case where the lens 1002 in FIG. 3 is connected to the imaging element having the spectral sensitivity in FIG. 4. In this case, the transmittance in the visible light region is low and the transmittance in the non-visible light region is high, so that if a color image is to be generated during infrared photography, the expected correction for infrared photography may not be able to handle it and the colors may be significantly distorted. Therefore, in this case, color output during infrared photography is disabled. In the example, the transmittance of the imaging optical system as shown in FIG. 3 is used to determine whether color output is possible during infrared photography, but it is also possible to determine whether color output is possible during infrared photography by registering an ID or the like that can identify an individual in advance. That is, the ID and whether color output is possible may be linked in a storage unit such as the ROM 106, and the ID information may be obtained from the imaging optical system to determine whether color output is possible. It is also possible for the user to send a notification that color output is forcibly enabled. Based on the determination result, the process proceeds to step S3 or step S4.

[0039] The determination between the first mode and the second mode will be described in more detail. Consider the case where the optical characteristics of the imaging optical system 101 are acquired by the lens information acquisition unit 112 in S2. At this time, it is assumed that the optical characteristics include at least the transmittance for infrared light (invisible light) as shown in FIG. 3. The mode determination unit 113, for example, determines whether the acquired transmittance is greater than a predetermined transmittance. If the transmittance for infrared light is greater than the predetermined transmittance, it determines that the color of the color image generated during infrared shooting will be distorted, and decides to generate and output the image in the first mode. On the other hand, if the transmittance for infrared light is smaller than the predetermined transmittance, it determines that the color of the color image generated during infrared shooting will not be distorted, and decides to generate and output the image in the second mode.

[0040] The predetermined transmittance is a reference transmittance and can be set by a user or a designer. In FIG. 3, the transmittance varies even within the non-visible light region, so the user or designer can also set which wavelength the transmittance is to be compared with. Alternatively, the first mode or the second mode may be determined using the average value of the transmittance in the non-visible light region.

[0041] In step S3, the WB control unit 114 performs WB control during infrared shooting based on the determination result of the mode determination unit 113. Specifically, in order to generate a color image, WB control is performed taking into account the influence of infrared light components. For example, control is performed to shift a control frame for drawing in white in advance taking into account the influence of infrared light components. As a preset function, a WB gain taking into account the influence of infrared light components may be used. Once the WB control during infrared shooting is determined, the process proceeds to step S6.

[0042] In step S4, the WB control unit 114 performs WB control during infrared shooting based on the determination result of the mode determination unit 113. Specifically, in order to generate a monochrome image, WB control during infrared shooting is performed. For example, it can be determined whether the monochrome image is to be generated with a black and white gradation or with a gradation of another single color. Once the monochrome image generation method has been determined, the process proceeds to step S5.

[0043] In step S5, the WB gain during infrared photography is fixed in the control determined by the WB control unit 114. When generating a monochrome image during infrared photography, the WB gain is not applied to the image, so the WB gain is fixed to prevent misidentification in a UI that displays the WB gain value. The fixed value can be freely determined to be 1x WB gain, 0x, an initial value, or the like. Misidentification can also be prevented by other methods, such as graying out the value display method. In that case, it is not necessarily necessary to fix the WB gain. Once the WB gain during infrared photography is determined, the process proceeds to step S6.

[0044] In step S6, image processing is performed based on information from the image processing unit 115 according to the shooting mode to generate an output image. The WB processing is different between the normal shooting mode and the infrared shooting mode. In contrast to the WB processing in the normal shooting mode, the WB processing in the infrared shooting mode performs control such as shifting the white pull-in frame in consideration of the influence of the infrared light component when generating a color image. When generating a monochrome image, the WB gain is fixed and an image with a black and white or selected monochromatic gradation is generated. Note that in the infrared shooting mode, when a color image can be generated, it is possible to generate a monochrome image by lowering the saturation as a post-processing. In addition, in an environment with low illuminance and almost no visible light component, color noise becomes noticeable when a color image is generated, so in such a case, the method may switch to a method of generating a monochrome image. In addition, even when the ratio of infrared light components becomes extremely large, such as when infrared lighting is irradiated, and the color of the color image during infrared shooting is greatly affected, the method may switch to a method of generating a monochrome image.

[0045] In an example of the imaging device in this embodiment, the processing during infrared photography can be controlled according to the type and characteristics of the connected imaging optical system. For example, for an imaging optical system whose operation is guaranteed in advance, a color image can be output during infrared photography, and for a lens whose operation is not guaranteed, a monochrome image is output during infrared photography instead of a color image. Alternatively, if the color image during infrared photography is expected to deviate significantly from the expected color due to the characteristics of the imaging optical system, a monochrome image is output during infrared photography. Here, by separating the processing between the WB control during infrared photography and the WB control during normal photography, only the WB control during infrared photography is changed according to the imaging optical system, and the WB control during normal photography is not affected. In a lens-interchangeable imaging device, it is common for a lens with guaranteed operation to be disclosed to the user, but if it is connectable, it is also possible to attach a handheld imaging optical system to the imaging device and take a picture. In such a case, there is a possibility that a performance that is significantly different from the function intended by the user may be erroneously provided. According to this embodiment, by providing a function for each imaging optical system, a function with incorrect performance is not provided, and it is possible to avoid variations in performance between imaging optical systems. Even in the case of a lens that is not interchangeable, the optical characteristics change depending on the use of an extender or the focal length, so it is possible to acquire these as lens information and control processing according to the lens information. The lens information is not limited to when the lens is correctly connected to the imaging device, and the user may directly provide the lens information. For example, in the case of a lens that is not guaranteed to operate, if the user provides the wavelength characteristics to the imaging device as lens information, processing can be controlled accordingly.

[0046] In this embodiment, the characteristics of the imaging optical system shown in Figures 3 and 4 are combined with the spectral sensitivity of the imaging element, but if the spectral sensitivity of the imaging element changes, the type of imaging optical system that can guarantee operation will change accordingly. For example, if the spectral characteristics of the imaging element change with temperature, it may be possible to control the imaging element so that a color image can be generated up to a certain temperature during infrared shooting, but a monochrome image is generated above or below a certain temperature. In addition, if the imaging element deteriorates over time and its spectral characteristics change, the control of the image generation mode may also change.

[0047] As described above, the imaging device of this embodiment can provide functions without affecting normal WB control and without large color variations depending on the characteristics of the imaging optical system.

[0048] <Second embodiment> The imaging device according to this embodiment will be described below. In the first embodiment, whether to generate a color image or a monochrome image during infrared imaging was determined based on information about the imaging optical system. In contrast, in this embodiment, in a mode for generating a color image during infrared imaging, i.e., in the second mode, the correction strength of WB control during infrared imaging is determined based on information about the imaging optical system.

[0049] As mentioned above, the transmittance and other characteristics of the imaging optical system cause variations in the color of color images captured during infrared photography. In imaging optical systems whose transmittance changes drastically depending on the wavelength band of light, or imaging optical systems that are not operationally guaranteed, we have prevented performance variations by making it possible to generate only monochrome images during infrared photography.

[0050] However, depending on the characteristics of the imaging optical system, there is a risk of some color variation. For example, in the case of infrared photography using a lens with a low transmittance, color variation may occur in the color image. In this embodiment, if a color image can be generated during infrared photography, the color variation for each imaging optical system can be reduced.

[0051] (Functional configuration) Fig. 5 is a block diagram showing the functional configuration of the imaging processing device according to this embodiment. Note that the device configuration of the imaging device according to this embodiment is the same as that of the first embodiment shown in Fig. 1(a), and therefore the description thereof will be omitted. In Fig. 5, the same functional units as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted. Below, the functional units having functions different from those in the first embodiment will be described.

[0052] The infrared correction intensity determination unit 201 determines the intensity of correction during infrared shooting in response to inputs from the lens information acquisition unit 112 and the mode determination unit 113, and outputs the determined intensity to the WB control unit 114. The intensity of correction during infrared shooting refers to changing the position of the white pull-in frame depending on the magnitude of the influence of the infrared light component when the infrared light component affects the output image, for example, in auto WB control. Alternatively, a fixed WB gain may be applied to each imaging optical system, such as in a preset function or manual WB function.

[0053] Fig. 6 is a flowchart showing the flow of image processing executed by the imaging device of this embodiment. An example of the imaging device according to this embodiment will be described below with reference to the flowchart of Fig. 6. Note that the same processing steps as those in the first embodiment are given the same reference numerals, and their description will be omitted. Processing steps different from those in the first embodiment will be described below.

[0054] In step S7, the correction intensity during infrared photography is determined according to the information on the influence of the infrared light component of the imaging optical system 101 acquired by the lens information acquisition unit 112 and the result of the mode determination unit 113. Specifically, if the mode determination unit 113 determines that a color image can be generated during infrared photography, the lens information acquisition unit 112 acquires information on the influence of the imaging optical system 101 on the infrared light component. Next, the WB correction intensity during infrared photography is determined according to the acquired information. The information on the influence of the infrared light component refers to characteristics such as the transmittance and reflectance of the infrared light component among the optical characteristics of the imaging optical system. The influence of the infrared light component may be assigned in advance by an ID that can identify an individual. When the correction intensity during infrared photography can be selected in multiple steps in advance, only the default correction intensity may be determined, or the multiple steps of correction intensity may be changed overall. Once the correction intensity is determined, the process proceeds to step S6.

[0055] The control of the imaging device according to the second embodiment and its effect will be described below. FIG. 7 is a diagram showing an example of the transmittance of an imaging optical system that can be connected to the imaging device in this embodiment. The lens 1101 drawn by a solid line is the same as the imaging optical system shown in FIG. 4, and in this embodiment, the 1011 lens is used as a reference lens to determine the correction strength during infrared shooting. Correction is performed to eliminate color variation from a color image during infrared shooting generated using the 1011 lens. For example, the lens of the dotted line 2102 in FIG. 7 is an imaging optical system with a higher transmittance on the non-visible light region side than the reference 1101 lens. In particular, 700 nm to 900 nm is a wavelength band close to the peak of the red pixel of the spectral curve 1001 in FIG. 3 showing the spectral sensitivity of the imaging element to the wavelength of light.

[0056] Therefore, when comparing the case where the lens 2102 is used with the case where the reference lens 1101 is used, the 2102 lens produces a reddish image in a color image captured in infrared light when the same intensity of WB correction is applied. More specifically, the 2102 lens transmits more light in the wavelength band that affects red pixels and delivers the light to the image sensor, so that a reddish image is produced because the correction intensity is the same as that of the reference lens 1101 and the image cannot be corrected correctly. In this case, by applying a stronger correction to the correction intensity of the reference lens 1101, an image of the same color as the color image captured in infrared light produced when the reference lens 1101 is used can be produced. As another example, the lens of the broken line 2103 in FIG. 7 has a transmittance that has a peak in a specific non-visible light region (around 850 nm) compared to the reference 1101 lens, and has a lower transmittance than the reference lens 1101 for the wavelength band close to the peak of the red pixel. That is, when the lens 2103 is used and the lens 1101 is used as the reference lens, the 2103 lens generates a more blue-green image when the same intensity of WB correction is applied to the color image captured in infrared light. This is because the 2103 lens is less likely to transmit light in the wavelength range that affects red pixels, and the correction intensity of the same as that of the reference lens 1101 is too strong, so that a blue-green image is generated. In this case, by applying a weaker correction to the correction intensity of the reference lens 1101, an image of the same color as the color image captured in infrared light generated when the reference lens 1101 is used can be generated. Since the spectral sensitivity characteristics of the image sensor are known in advance, the same image as that of the reference lens can be obtained by strengthening or weakening the correction intensity when the reference imaging optical system is used based on the characteristics of the imaging optical system to be used. In other words, the WB correction intensity is determined based on the information about the imaging optical system 101 acquired by the lens information acquisition unit 112. More specifically, when the transmittance of the connected imaging optical system 101 is greater than a predetermined transmittance (for example, the transmittance of a reference lens), the WB correction strength is increased, whereas when the transmittance of the connected imaging optical system 101 is less than the predetermined transmittance, the WB correction strength is decreased.In addition, since the transmittance and reflectance of the infrared light component are in a relative relationship, when determining the WB correction strength based on the reflectance, the opposite process to that for the transmittance can be performed: if the reflectance is smaller than a predetermined reflectance, the WB correction strength is increased, and if the reflectance is larger than the predetermined reflectance, the WB correction strength is decreased.

[0057] The degree to which the correction strength is increased or decreased may be determined based on, for example, the difference or ratio between a predetermined transmittance (reflectance) and the acquired transmittance (reflectance). That is, the greater the transmittance of the imaging optical system 101, the greater the WB correction strength is increased, and the smaller the transmittance of the imaging optical system 101, the greater the WB correction strength is decreased.

[0058] According to this embodiment, it is possible to reduce the variation in color of an image that occurs depending on the characteristics of the imaging optical system used.

[0059] The effect of the infrared light transmittance on the color image as described above can be reduced by inserting an infrared cut filter onto the optical axis. However, this has the disadvantage that the amount of light cannot be increased in low-illumination environments, so it is preferable to avoid inserting an infrared cut filter unless the environment is sufficiently illuminant.

[0060] Fig. 8 is an example diagram showing the difference in color images generated during infrared photography depending on the lens used. Images generated according to the correction strength for the reference lens 1101 in Fig. 7 are also shown, with the correction strength increasing from the upper left to the lower right in Fig. 8. In Fig. 8, reddish and blue-green colors are expressed by the amount of black and white. Originally displayed as a color image, when the WB is near the median, a weaker correction strength generates a redder image, and a stronger correction strength generates a blue-green image.

[0061] 2201 is an image generated when the reference lens 1101 in FIG. 7 is used when outputting a color image during infrared photography, and is the image with the most appropriate WB in a certain environment. 2202 is a reddish image generated using the lens 2102 in FIG. 7 when no correction is made for each imaging optical system when outputting a color image during infrared photography. 2103 is a blue-greenish image generated using the lens 2103 in FIG. 7 when no correction is made for each imaging optical system when outputting a color image during infrared photography. 2204 is a reddish image with the weakest correction strength when corrected using the reference lens 1101 when outputting a color image during infrared photography. 2205 is a blue-greenish image with the strongest correction strength when corrected using the reference lens 1101 when outputting a color image during infrared photography.

[0062] In Fig. 8, when the image 2204 with the weakest correction strength is defined as correction level 0, the image 2201 corrected with the default strength with the correct WB is set to correction level 4, and the image 2205 with the strongest correction is set to correction level 7, for a total of 8 correction strength levels. However, the number of correction strength levels can be arbitrarily determined, such as two levels (weak and strong), one level (default only), or more.

[0063] If the correction intensity is not changed according to the characteristics of the imaging optical system, in the example of the lens used in Fig. 7 as described above, the colors of the images generated with the default correction for each imaging optical system will vary, as in images 2202 and 2203 in Fig. 8. When lens 2102 in Fig. 7 is used, image 2202 in Fig. 8, which corresponds to correction level 2 at the correction intensity for reference lens 1101, is generated, so the default correction intensity is strengthened by two steps. As a result, the default correction intensity when imaging optical system 2102 is used is stronger, so a default image can be output with the same correction level 4.

[0064] When the lens 2103 in FIG. 7 is used, an image 2203 in FIG. 8 is generated, which corresponds to correction level 6 at the correction strength with the reference lens 1101, so the default correction strength is weakened by two steps. In this way, since the default correction strength when the imaging optical system 2103 is used is weakened, a default image can be output at the same correction level 4. Therefore, by changing the correction strength for each imaging optical system, a default correction strength corresponding to each imaging optical system is determined, and a color image with reduced color variation compared to the default correction is generated even if the characteristics of the lens used change.

[0065] In this embodiment, the default value is set to be near the median value of the correction level, but any level may be set as the default. That is, instead of changing the default correction intensity internally, the color variation may be reduced by changing the correction level. Specifically, the default correction level when the 2202 lens is used is set to correction level 6, which is two steps higher than the reference. Alternatively, the default correction level when the 2203 lens is used is set to correction level 2, which is two steps lower than the reference. This makes it possible to suppress color variation in color images during infrared shooting. In such a case, since there is a risk that the correction range will be narrowed to one side, the correction range may be limited to a side close to the default correction level. In this way, it is possible to avoid not being able to process as intended when the correction level is changed. In addition to the default correction intensity, the correction intensity associated with the level may be changed according to the characteristics of the imaging optical system for each correction level. This allows the user to select the correction level with the same feeling even if the lens used is changed. In addition, when the correction level is automatically controlled, the correction level switching threshold may be changed according to the characteristics of the imaging optical system. This allows a color image with automatically suppressed color variation to be generated even when the color image during infrared shooting differs depending on the characteristics of the imaging optical system.

[0066] As described above, according to the imaging device of this embodiment, by changing the WB correction strength taking into account the influence of the infrared light component for each imaging optical system used, it is possible to reduce color variation in the color image generated during infrared shooting.

[0067] Although the image processing according to the above embodiment is applied to an imaging device, the image processing according to the present embodiment may be realized by using an information processing device (computer) such as a personal computer or a smartphone connected to the imaging device as the image processing device. In this case, the imaging device outputs to the computer the raw data captured by the imaging unit, the shooting parameters indicating the exposure time, frame rate, exposure setting value, etc., and information indicating the presence or absence of the influence of infrared light components on the color of the input image. The CPU in the computer receives the image signal output from the imaging device, thereby fulfilling the function of the first acquisition unit described in the first embodiment.

[0068] Information about the imaging optics may be input by a user, and image processing similar to that described in the previous embodiment is performed in a computer, which in this example executes the program code of the software that realizes the image processing of this embodiment.

[0069] The imaging device according to the present embodiment may be configured such that at least a part of the processing by the CPU is executed by dedicated hardware, such as an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0070] The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0071] 101 Imaging optical system 102 Optical Filters 103 Image sensor 104 CPU 105 RAM 106 ROM 107 Bus 111 Optical filter control section 112 Lens information acquisition unit 113 Mode determination unit 114 WB control section 115 Image Processing Unit

Claims

1. A first acquisition unit that acquires information regarding the imaging optical system; A second acquisition unit that acquires an image signal from the imaging element; an image processing unit that generates an image in a first mode that generates an achromatic or monochrome image based on the image signal, or in a second mode that generates an image including a plurality of chromatic colors based on the image signal, The image processing device according to claim 1, wherein the image processing unit determines in which of the first mode and the second mode an image is to be generated, based on the information.

2. The image processing device according to claim 1 , wherein the imaging optical system is connected in an exchangeable manner, and the first acquisition unit acquires information about the connected imaging optical system.

3. 3. The image processing device according to claim 1, wherein the image processing unit switches between the first mode and the second mode based on a ratio of a visible light component to an infrared light component of the image signal.

4. 3. The image processing device according to claim 1, wherein the image processing section fixes a white balance gain for the image signal in the first mode.

5. 3. The image processing apparatus according to claim 1, wherein the image processing unit performs white balance correction on the image signal in the second mode.

6. The image processing apparatus according to claim 5 , wherein the image processing unit determines a correction intensity of the correction based on the information acquired by the first acquisition unit.

7. The image processing apparatus according to claim 5 , wherein the image processing unit changes a range of the correction strength of the correction based on the information acquired by the first acquisition unit.

8. the information includes optical characteristics of the imaging optical system; The image processing apparatus according to claim 5 , wherein, in the second mode, the image processing unit determines a correction intensity of the correction based on the optical characteristics.

9. the optical characteristics include a transmittance of the imaging optical system with respect to infrared light, The image processing apparatus according to claim 8 , wherein the image processing unit determines a correction intensity of the correction based on the transmittance acquired by the first acquisition unit.

10. The image processing apparatus according to claim 9 , wherein the image processing unit increases a correction strength of the correction when the transmittance is greater than a predetermined transmittance.

11. The image processing device according to claim 9 , wherein the image processing unit reduces a correction strength of the correction when the transmittance is smaller than a predetermined transmittance.

12. the information includes optical characteristics of the imaging optical system; 3. The image processing device according to claim 1, wherein the image processing unit determines whether an image is to be generated in the first mode or the second mode based on the optical characteristics.

13. the optical characteristics include a transmittance of the imaging optical system with respect to infrared light, The image processing device according to claim 12 , wherein the image processing unit determines in which of the first mode and the second mode an image is to be generated, based on the transmittance acquired by the first acquisition unit.

14. The image processing device according to claim 13 , wherein the image processing unit generates an image in the first mode when the transmittance is greater than a predetermined transmittance.

15. The image processing device according to claim 13 , wherein the image processing unit generates an image in the second mode when the transmittance is smaller than a predetermined transmittance.

16. 3. The image processing apparatus according to claim 1, wherein the information includes identification information for identifying a type of the imaging optical system.

17. A storage unit that stores optical information of each of the plurality of imaging optical systems, The image processing apparatus according to claim 16 , wherein the first acquisition unit acquires optical information of the connected imaging optical system from the storage unit based on the identification information.

18. A first acquisition step of acquiring information about an imaging optical system; a second acquisition step of acquiring an image signal from the imaging element; and an image processing step of generating an image in a first mode for generating an achromatic or monochrome image based on the image signal or in a second mode for generating an image including a plurality of chromatic colors based on the image signal. The image processing method according to claim 1, wherein in the image processing step, it is determined whether an image is to be generated in the first mode or the second mode based on the information.

19. A program for causing a computer to execute the image processing method according to claim 18.

20. 20. A computer-readable storage medium storing the program according to claim 19.

Images