Imaging control device, imaging device and control method thereof

Abstract

To acquire invisible light image data while suppressing power consumption in an imaging apparatus that can acquire visible light image data and non-visible light image data.SOLUTION: An imaging apparatus includes imaging means 100 and 101 that generate visible light image data and non-visible light image data by imaging, processing means 103, 107 that perform image processing using at least one of visible light image data and non-visible light image data, determination means 110 to 112, 117, 118, 119 that determine an imaging scene, and control means 117 that controls whether to generate invisible light image data by the imaging means and to perform image processing using the invisible light image data in the processing means, according to a determination result of the imaging scene.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device capable of acquiring visible light images and non-visible light images.

Background Art

[0002] Non-visible light image data obtained by imaging non-visible light such as near-infrared light can be used for image processing on visible light image data obtained by imaging visible light, for example, noise reduction, contrast improvement, haze removal, and shake correction processing in dark places, hazy or foggy states, and long-distance imaging. However, in order to obtain visible light image data and non-visible light image data by imaging, an image sensor for visible light imaging and an image sensor for non-visible light imaging are required. Although there are image sensors that can perform visible light imaging and non-visible light imaging simultaneously, for each of the visible light image data and the non-visible light image data, processing such as pixel correction, black level correction, and defect correction caused by the image sensor is required. Therefore, the power consumption is larger than that of an imaging device that performs only visible light imaging. In addition, when imaging is performed using a light source that emits non-visible light, the power consumption increases to energize the light source.

[0003] Patent Document 1 discloses an imaging device that allows a user to select a switch from a state of acquiring visible light and non-visible light image data to a state of acquiring only visible light image data when it is determined that the imaged scene has changed. Patent Document 2 discloses an imaging device that controls the light amounts of a visible light source and a non-visible light source according to the color temperature of a subject.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the imaging device disclosed in Patent Document 1 determines changes in the imaging scene based on the temporal change in color. Therefore, invisible light image data may be acquired even in imaging scenes that do not require invisible light image data. Furthermore, the imaging device disclosed in Patent Document 2 adjusts the light intensity of the invisible light source based on the color temperature of the subject, so there is a risk of using the invisible light source even in imaging scenes where it is not necessary. In this case, the power consumption of the imaging device will increase.

[0006] The present invention provides an imaging device and a control method therefor that can acquire both visible light image data and invisible light image data, and that can acquire invisible light image data while suppressing power consumption. [Means for solving the problem]

[0007] An imaging device as one aspect of the present invention is characterized by comprising: an imaging means for generating visible light image data and invisible light image data by imaging; a processing means for performing image processing using at least one of the visible light image data and invisible light image data; a determination means for determining the imaging scene; and a control means for controlling whether or not to perform the generation of invisible light image data by the imaging means and the image processing using the invisible light image data in the processing means, according to the determination result of the imaging scene.

[0008] Another aspect of the present invention, the control method, is applied to an imaging apparatus having an imaging means that generates visible light image data and invisible light image data by imaging, and a processing means that performs image processing using at least one of the visible light image data and invisible light image data. The control method is characterized by having the steps of determining an imaging scene and controlling whether or not to perform the generation of invisible light image data by the imaging means and the image processing using the invisible light image data by the processing means, according to the determination result of the imaging scene. A program that causes the computer of the imaging apparatus to perform processing according to the above control method also constitutes another aspect of the present invention. [Effects of the Invention]

[0009] According to the present invention, in an imaging device capable of acquiring visible light data and invisible light image data, invisible light image data can be acquired while suppressing power consumption. [Brief explanation of the drawing]

[0010] [Figure 1] A block diagram showing the configuration of the imaging device in Example 1. [Figure 2] A flowchart illustrating the imaging process in the imaging device of Example 1. [Figure 3] A flowchart illustrating the process for determining whether to use or not use a non-visible light image in Example 1. [Figure 4] A diagram showing whether or not invisible light images can be used and their applications in Example 1. [Figure 5] A figure showing an example of the display image in Example 1. [Figure 6] A block diagram showing the configuration of the imaging device in Example 2. [Figure 7] A diagram showing the arrangement of the image sensor filters in Example 2. [Figure 8] A figure showing an example of the display image in Example 3. [Figure 9] A figure showing an example of the output image in Example 4. [Figure 10] A figure showing an example of the display image in Example 5. [Modes for carrying out the invention]

[0011] The embodiments of the present invention will now be described with reference to the drawings. The imaging devices that are embodiments of the present invention include not only digital cameras, which will be described later, but also imaging devices that are installed in various electronic devices. Various electronic devices include computer equipment (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, robots, drones, in-vehicle cameras, and surveillance cameras, etc. [Examples]

[0012] FIG. 1 shows the configuration of a digital camera (hereinafter simply referred to as a camera) according to Embodiment 1. The camera includes a visible light imaging device 100, a non-visible light imaging device 101, a first image processing unit 106, a second image processing unit 102, a control bus 104, a data bus 105, a memory control unit 113, a memory 114, a non-volatile memory control unit 115, and a non-volatile memory 116. Further, the camera includes a main control unit 117, a camera state acquisition unit 118, and a distance information acquisition unit 119. Further, the camera includes a display control unit 120, a display unit 121, a non-visible light source 122, a communication unit 123, and a compression / expansion unit and an external recording unit (not shown).

[0013] The visible light imaging device (first imaging means for visible light) 100 is a photoelectric conversion device such as a CCD sensor or a CMOS sensor, and converts visible light (subject image) received through red (R), green (G), and blue (B) color filters into an electrical signal to generate pre-correction visible light image data. The R, G, and B color filters are arranged in a predetermined pattern for a plurality of pixels on the imaging surface of the visible light imaging device 100, have a peak in the light transmittance in the wavelength band of each color, and block the transmission of light in other wavelength bands.

[0014] The non-visible light imaging device (second imaging means for non-visible light) 101 is a photoelectric conversion device such as a CCD sensor or a CMOS sensor, and converts a non-visible light subject image received through a near-infrared filter into an electrical signal to generate pre-correction non-visible light image data. The near-infrared filter is arranged for each of a plurality of pixels on the imaging surface of the non-visible light imaging device 101, has a peak in the light transmittance in the near-infrared wavelength band, and blocks the transmission of light in other wavelength bands. Note that the non-visible light imaging device 101 may image non-visible light in a wavelength band other than the near-infrared wavelength band. The visible light imaging device 100 and the non-visible light imaging device 101 constitute the imaging means.

[0015] The second image processing unit 102 is composed of an invisible light image correction unit 103 and a DMAC (Direct Memory Access controller) not shown. The invisible light image correction unit 103 performs image processing such as pixel correction, black level correction, shading correction, and defect correction on the pre-correction invisible light image data generated by the invisible light image sensor 101, and generates invisible light image data. The DMAC of the second image processing unit 102 is connected to the data bus 105, writes the invisible light image data to the memory 114 via the memory control unit 113, or reads the pre-correction invisible light image data on the memory 114 and outputs it to the invisible light image correction unit 103 via the memory control unit 113.

[0016] The first image processing unit 106 is composed of a visible light image correction unit 107, a development unit 108, a synthesis unit 109, a recognition unit 110, an image integration unit 111, a vector detection unit 112, and a DMAC not shown. The DMAC is connected to the data bus 105, and writes the data processed by the visible light image correction unit 107, the development unit 108, the synthesis unit 109, the recognition unit 110, the image integration unit 111, and the vector detection unit 112 to the memory 114 via the memory control unit 113. Also, the DMAC reads the data on the memory 114 and outputs it to the visible light image correction unit 107, the development unit 108, the synthesis unit 109, the recognition unit 110, the image integration unit 111, and the vector detection unit 112 via the memory control unit 113. The first image processing unit 106 and the second image processing unit 102 constitute processing means.

[0017] The visible light image correction unit 107 performs image processing such as pixel correction, black level correction, shading correction, and defect correction on the pre-correction visible light image data generated by the visible light image sensor 100, and generates visible light image data.

[0018] The developing unit 108 performs image processing on visible light image data and visible light image data stored in memory 114, including white balance adjustment (AWB), chromatic aberration correction, gamma correction, brightness and color generation processing, geometric deformation, and noise reduction. In addition, the developing unit 108 performs image processing on non-visible light image data stored in memory 114 in step S303 of Figure 3, which will be described later, including gamma correction, brightness and color generation processing, geometric deformation, and noise reduction.

[0019] The image synthesis unit 109 synthesizes the visible light image data and the invisible light image data in step S303 of Figure 3, and performs image processing such as noise reduction, contrast improvement, and shake correction during the synthesis. In this embodiment, the visible light image sensor 100 and the invisible light image sensor 101 are driven simultaneously to acquire the uncorrected visible light image data and uncorrected invisible light image data at the same time for noise reduction and contrast improvement. However, the visible light image sensor 100 and the invisible light image sensor 101 may acquire the uncorrected visible light image data and uncorrected invisible light image data at different times, and these image data may be used after image alignment to compensate for the time difference in acquisition. In this way, the image processing performed by the image synthesis unit 109 can also be applied to moving image data. Furthermore, shake correction will be explained on the premise that the shutter speeds of the visible light image sensor 100 and the invisible light image sensor 101 are different.

[0020] In noise reduction, for example, a Dual Bilateral Filter (DF) is used to remove noise from visible light image data by using non-visible light image data. Since the DF uses all the edge information from both the non-visible and visible light image data, edges that are unclear in the visible light image data are interpolated, improving edge preservation and noise reduction capabilities.

[0021] For contrast improvement, for example, histograms of pixel values ​​are obtained for both visible light image data and non-visible light image data. First, if the histogram of visible light image data (with pixel values ​​on the horizontal axis and frequency on the vertical axis) is distributed over a narrower range than the histogram of non-visible light image data, that is, if the proportion m in equation (1) below is less than 1, the contrast is judged to be low and contrast improvement is performed.

[0022] m = histogram range of visible light image data (horizontal axis) Histogram range of invisible light image (horizontal axis) (1) Next, in contrast improvement, the luminance signal is extracted from the visible light image data, and then the luminance signal is subjected to a wavelet transform to separate it into low-frequency and high-frequency components. Then, the luminance values ​​of the low-frequency component and the original (visible light image data) low-frequency component are calculated using equation (2), with the histogram of the magnitude of the spatial gradient of the low-frequency component being aligned with the histogram of the magnitude of the gradient of the non-visible light image data.

[0023] (Low-frequency component with aligned histogram × m + original low-frequency component × (1-m)) / 2 (2) Furthermore, for contrast improvement, the luminance value of the high-frequency components is calculated using equation (3).

[0024] (High-frequency component of invisible light image data × m + original high-frequency component × (1-m)) / 2 (3) For contrast improvement, image data with improved contrast is generated by performing an inverse wavelet transform on the luminance values ​​of the low-frequency components and the luminance values ​​of the high-frequency components.

[0025] In image stabilization, for example, if the shutter speed of imaging with the visible light image sensor 100 is slow and there is a risk of image stabilization due to camera shake caused by hand shake, the shutter speed of imaging with the non-visible light image sensor 101 is increased, and multiple frames of non-visible light image data are acquired. Then, a blur kernel is estimated from the multiple frames of non-visible light image data, and image stabilization is corrected by performing deconvolution on the visible light image data using the estimated blur kernel.

[0026] Furthermore, for noise reduction, contrast improvement, and shake correction performed in the synthesis unit 109, methods other than those described above may be employed.

[0027] The recognition unit 110 performs subject recognition by detecting one or more predetermined candidate subject regions (candidate regions) from the input image data of one frame using deep learning (DL). For each candidate region, the recognition unit 110 associates the input image data with its position and size within the frame, the object class indicating the type of candidate subject (car, airplane, bird, insect, human body, head, pupil, cat, dog, etc.), and its confidence level. It also counts the number of regions detected for each object class. Furthermore, it recognizes the imaging scene of the input image data (sky, night view, landscape, sports, portrait, food, race, etc.) and associates its confidence level with the input image data.

[0028] Such a recognition unit 110 can detect candidate regions using known techniques for detecting feature regions such as the facial regions of people or animals. For example, the recognition unit 110 may be configured as a class classifier trained using training data. There are no particular restrictions on the classification algorithm. The recognition unit 110 can be realized by training a classifier that implements multi-class logistic regression, support vector machines, random forests, or neural networks.

[0029] Furthermore, when performing subject recognition using non-visible light image data, the recognition rate may decrease if the learning model of the recognition unit 110 is a learning model for visible light images; therefore, it may be changed to a learning model for non-visible light images. A learning model is a model that, in response to an input, evaluates the content of that input and outputs that as an output value. It is a model that is learned using a dataset for a certain purpose and is obtained as a result of that learning.

[0030] The recognition unit 110 does not necessarily use DL. For example, it may determine that the image scene is a night scene if the overall brightness of the image is dark and there are multiple areas with high brightness.

[0031] The recognition unit 110 also performs subject tracking using DL (DL tracking) or subject tracking without using DL (non-DL tracking). DL tracking estimates the position and size of the subject region to be tracked using a pre-trained multilayer neural network including convolutional layers. More specifically, it has a function to extract feature points and feature quantities contained in the feature points for each object class that may be the target, and a function to associate the extracted feature points between frames. Such DL tracking estimates the position and size of the subject region to be tracked in the current frame from the feature points of the current frame that are associated with the feature points of the subject region to be tracked in past frames. Then it outputs the position, size, and confidence score of the subject region to be tracked estimated for the current frame.

[0032] The confidence score indicates the reliability of the feature point mapping between frames, i.e., the reliability of the estimation result for the subject region being tracked. If the confidence score indicates low reliability of the feature point mapping between frames, it suggests that the subject region estimated in the current frame may be a region relating to a different subject than the subject region being tracked in previous frames.

[0033] On the other hand, non-DL tracking estimates the subject region to be tracked in the current frame using methods that do not employ deep learning. In this case, non-DL tracking estimates the subject region to be tracked in past frames using pattern matching as a template, but other methods may also be used.

[0034] The image integration unit 111 divides the input image data into multiple rectangular regions, integrates the pixel values ​​for each region, and outputs the integration result. The integration result is used, for example, for automatic exposure (AE) or white balance adjustment by the development unit 108.

[0035] The vector detection unit 112 detects the motion vector of the subject from the previous frame and the current frame in the input image data using a block matching method or the like. However, other methods may be used to detect the motion vector.

[0036] The recognition unit 110, image integration unit 111, and vector detection unit 112 receive either visible light image data or invisible light image data, or both, as input image data. The recognition unit 110, image integration unit 111, and vector detection unit 112 may select the input image data from visible light image data and invisible light image data depending on the imaging scene recognized by the recognition unit 110.

[0037] The control bus 104 is used when the main control unit 117 accesses the first image processing unit 106, the second image processing unit 102, the visible light image sensor 100, the non-visible light image sensor 101, the display control unit 120, the memory control unit 113, and the non-volatile memory control unit 115. It is also used when accessing the camera status acquisition unit 118, the distance information acquisition unit 119, the non-visible light source 122, and the communication unit 123.

[0038] The data bus 105 is used to transfer data between the memory 114 or non-volatile memory 116 and the main control unit 117, the first image processing unit 106, the second image processing unit 102, the display control unit 120, the camera status acquisition unit 118, the distance information acquisition unit 119, and the communication unit 123.

[0039] The memory control unit 113 writes data to the memory 114 and reads data from the memory 114 in response to instructions from the main control unit 117, the first image processing unit 106, the second image processing unit 102, the display control unit 120, the camera status acquisition unit 118, the distance information acquisition unit 119, and the communication unit 123. The memory 114 is composed of DRAM or the like and stores a predetermined number of still image data, a predetermined amount of moving image data and audio data, constants for the operation of the main control unit 117, programs, etc.

[0040] The non-volatile memory control unit 115 writes data to and reads data from the non-volatile memory 116 in response to instructions from the main control unit 117. The non-volatile memory 116 is an electrically erasable and recordable memory, such as an EEPROM. The non-volatile memory 116 stores constants for the operation of the main control unit 117, programs, etc.

[0041] The main control unit 117, acting as a control means, is composed of a microcomputer and is responsible for the overall control of the camera. That is, it issues various instructions to each functional block within the camera and executes various processes. The main control unit 117 controls the first image processing unit 106, the second image processing unit 102, the visible light image sensor 100, the non-visible light image sensor 101, the display control unit 120, the memory control unit 113, and the non-volatile memory control unit 115, which are connected via the control bus 104. Furthermore, it controls the camera state acquisition unit 118, the distance information acquisition unit 119, the non-visible light source 122, and the communication unit 123. The main control unit 117 executes each process according to the program recorded in the non-volatile memory 116.

[0042] Furthermore, the main control unit 117 determines the imaging scene that the camera is attempting to capture based on the outputs from the recognition unit 110, image integration unit 111, vector detection unit 112, camera state acquisition unit 118, and distance information acquisition unit 119, as will be described later. The determination means is composed of the recognition unit 110, image integration unit 111, vector detection unit 112, camera state acquisition unit 118, distance information acquisition unit 119, and main control unit 117.

[0043] The camera status acquisition unit 118 acquires the camera status and operating settings. The camera status refers to the camera's attitude and shake, etc., acquired by a gyro sensor (not shown). The camera operating settings are made by the user by displaying a menu screen on the display unit 121, and include settings such as shutter speed, AE ON / OFF, and AWB / manual WB.

[0044] The distance information acquisition unit 119 acquires information on the subject distance, which is the distance from the camera to the subject. The subject distance can be calculated, for example, by measuring the Time of Flight (ToF), that is, the time it takes for near-infrared light to be shone from the camera onto the subject and for the reflected light to return to the camera, and multiplying this time by the speed of light. Other methods may also be used to acquire the subject distance.

[0045] The display control unit 120 controls the display of image data and UI (user interface) on the display unit (monitor) 121. The display unit 121 is composed of a liquid crystal panel, an organic EL panel, etc., and displays image data, UI, etc. The display control unit 120 and the display unit 121 constitute the display means.

[0046] The invisible light source 122 is composed of invisible light LEDs, etc., and receives control from the main control unit 117 regarding the amount of light emitted, the timing of light emission, the duration of light emission, and the irradiation position, and irradiates the subject with invisible light. The irradiation position is, for example, the position of the person when a person is recognized in the imaging data and light is irradiated onto that person.

[0047] The communication unit 123 transmits image data processed by the first image processing unit 106 and the second image processing unit 102, unprocessed visible light RAW data and invisible light RAW data from the development unit 108, and compressed image data compressed into JPEG format, MPEG format, etc. to the outside of the camera. Visible light / invisible light RAW data is image data that has been processed or is unprocessed by the visible light / invisible light image correction units 107 and 103. The communication unit 123 exchanges image data and other data with external devices via communication. The communication unit 123 communicates with external devices using either wired communication, wireless communication, or both.

[0048] In addition, although not shown in the diagram, the camera includes a compression / decompression unit that compresses each image data into JPEG or MPEG format, and decompresses the compressed data, as well as an external recording unit that records the image data to an external recording medium.

[0049] The flowchart in Figure 2 shows the imaging process that the main control unit 117 executes according to the program after the camera is powered on. After powering on, the main control unit 117 performs predetermined initialization processing and then proceeds to the imaging process.

[0050] In step S201, the main control unit 117 retrieves the imaging mode set in the camera, which is stored in the memory 114. The imaging modes include UI mode, still image recording mode (single shot / continuous shot mode), and video recording mode.

[0051] In step S202, the main control unit 117 determines whether the imaging mode acquired in step S201 is UI mode or not. If it is UI mode, the unit proceeds to step S203; otherwise, it proceeds to step S204.

[0052] In step S203, the main control unit 117 displays a UI on the display unit 121 and writes the settings information made by the user through the UI to the memory 114. Then, it exits the UI mode and proceeds to step S211. The settings referred to here include, for example, settings for camera operation and settings for still image processing (step S206), still image LV processing (step S207), video recording processing (step S209), and video LV processing (step S210), which will be described later. Specifically, these include ON / OFF of the camera's power saving mode, automatic / manual white balance, continuous shooting mode and number of continuous shots in still image processing, and the number of recording pixels and frame rate in video recording processing. Furthermore, using the invisible light source 122 and invisible light image sensor 101 to obtain invisible light image data increases power consumption compared to when they are not used. For this reason, the non-use of invisible light image data may always be set in power saving mode, or the user may set the use / non-use of power-saving invisible light image data through the UI.

[0053] Furthermore, if the noise reduction and shake correction levels in the composite unit 109 can be set to weak, medium, or strong, the use of invisible light image data may be enabled when the setting is strong via the UI. Also, in the LV processing for displaying the live view (LV) image on the display unit 121, the use / non-use of invisible light image data may be enabled via the UI. Moreover, if it is determined in step S302 of Figure 3 (described later) that invisible light image data should be used, the user may be able to set via the UI whether or not to save the invisible light RAW data as invisible light image data in the memory 114. By enabling such settings, if the user does not later use the invisible light RAW data that has not undergone noise reduction, contrast improvement, and shake correction in the composite unit 109, the power required to save the invisible light RAW data and the capacity of the memory 114 can be reduced.

[0054] In step S204, the main control unit 117 determines whether the imaging mode acquired in step S201 is still image recording mode or not. If it is still image recording mode, the unit proceeds to step S205; otherwise, it proceeds to step S208.

[0055] In step S205, the main control unit 117 determines whether the user has operated a shutter button (not shown) to instruct still image capture. If so, the unit proceeds to step S206; otherwise, it proceeds to step S207.

[0056] In step S206, the main control unit 117 performs still image recording processing. In still image recording processing, the first image processing unit 106 (development unit 108 or synthesis unit 109) generates still image data as output image data, compresses the still image data in JPEG format or the like, and records it on an external recording medium. When the still image recording processing is completed, the process proceeds to step S211.

[0057] In step S207, the main control unit 117 performs still image LV processing. In still image LV processing, the first image processing unit 106 generates LV image data as output image data, and displays this LV image data on the display unit 121 at the frame rate set in the UI processing in step S203. In addition, during still image LV processing, the drive timing of the visible light image sensor 100 and the non-visible light image sensor 101, as well as the visible light image correction unit 107 and the non-visible light image correction unit 103 are controlled. When still image LV processing is completed, the process proceeds to step S211.

[0058] In step S208, the main control unit 117 determines whether the imaging mode acquired in step S201 is video recording mode or not. If it is video recording mode, the unit proceeds to step S209; otherwise, it proceeds to step S210.

[0059] In step S209, the main control unit 117 performs video recording processing. In video recording processing, the first image processing unit 106 generates video data as output image data by generating image data for each frame. Then, this video data is compressed in MPEG format or the like and recorded on an external recording medium. When the video recording processing is completed, the process proceeds to step S211.

[0060] In step S210, the main control unit 117 performs video LV processing. In video LV processing, similar to still image LV processing, the first image processing unit 106 generates LV image data as output image data, and displays this LV image data on the display unit 121 at the frame rate set in the UI processing in step S203. In addition, in video LV processing, the drive timing of the visible light image sensor 100 and the invisible light image sensor 101, as well as the visible light image correction unit 107 and the invisible light image correction unit 103 are controlled. When video LV processing is completed, the process proceeds to step S211.

[0061] In step S211, the main control unit 117 determines whether the power switch (not shown) has been operated by the user to turn off the camera's power. If the power is turned off, the process proceeds to step S212; otherwise, it returns to step S201.

[0062] In step S212, the main control unit 117 performs the termination process for the camera operation. In this termination process, the non-volatile memory 116 is updated with the configuration information written to memory 114 in the UI processing in step S203.

[0063] The flowchart in Figure 3 shows the setting process for using or not using invisible light image data, which is performed by the main control unit 117 in the still image recording process (step S206), still image LV processing (step S207), video recording process (step S209), and video LV processing (step S210) in Figure 2. Using or not using invisible light image data, in other words, is the execution or non-execution of imaging by the invisible light image sensor 101 and processing by the second image processing unit 102.

[0064] In step S301, the main control unit 117 acquires invisible light image usage determination information. Specifically, the main control unit 117 sets (updates) the invisible light image usage determination information to use invisible light image data if the recognition result of the recognition unit 110 is a night view or if the recognition scene is set as a landscape or night view in the UI processing in step S203. Otherwise, the invisible light image usage determination information is set so that invisible light image data is not used.

[0065] Furthermore, the main control unit 117 determines the type of light source in the imaging scene based on the integration result from the image integration unit 111. A blackbody radiation curve can be used to determine the type of light source. In this case, if it is difficult to determine the type of light source because the color temperature of the mercury lamp is close to the color temperature of the grass, that is, if it is determined that the imaging scene has a light source that is not suitable for the main control unit, the invisible light image usage determination information is set to use invisible light image data. Otherwise, the invisible light image usage determination information is set not to use invisible light image data.

[0066] Furthermore, the main control unit 117, based on the brightness information calculated from the region-specific integration results of the image integration unit 111, sets invisible light image usage determination information to use invisible light image data if it determines that the imaging scene requires increasing the ISO sensitivity through AE control. Otherwise, it sets invisible light image usage determination information to not use invisible light image data.

[0067] Furthermore, if the main control unit 117 determines from the detection results by the gyro sensor of the camera state acquisition unit 118 that camera shake is occurring and that the imaging scene is dark, it sets the invisible light image usage determination information to use invisible light image data. Otherwise, it sets the invisible light image usage determination information to not use invisible light image data. Dark imaging scenes are determined when the recognition result of the imaging scene by the recognition unit 110 is a night scene, when the imaging scene is set to "night scene" in the UI processing, or when the imaging scene is determined to require increasing the ISO sensitivity in AE control based on the brightness information calculated from the integration result by the image integration unit 111.

[0068] Furthermore, the non-visible light image usage determination information includes not only whether non-visible light image data is used or not, but also the recognition result of the recognition unit 110, the imaging scene setting in the UI processing, the light source determination result, brightness information, AE control (ISO sensitivity) information, camera status (shake, etc.), motion vector detection result, and subject distance information.

[0069] In step S302, the main control unit 117 determines whether or not to use invisible light image data based on the invisible light image usage determination information obtained in step S301. If it is to use the image data, it proceeds to step S303; otherwise, it proceeds to step S304.

[0070] In step S303, the main control unit 117 turns on the drive of the invisible light image sensor 101 to enable the acquisition of invisible light image data, and transitions the camera to a state where it can process the invisible light image data. At this time, the main control unit 117 controls the illumination / de-illumination of the subject with invisible light from the invisible light light source 122 according to the invisible light image usage determination information. Then the process ends.

[0071] In step S304, the main control unit 117 turns off the drive of the invisible light image sensor 101, making it impossible to acquire invisible light image data, and transitions the camera to a state where it does not process invisible light image data. Then, this process ends.

[0072] Figure 4 summarizes the use (○) and non-use (no mark) of invisible light image data for each condition (determination method), and the applications when used. In the applications where it is used, contrast improvement, noise reduction, and shake correction are processes performed by the synthesis unit 109 using invisible light image data. WB determination means that invisible light image data is used for light source determination. Switching of recognition unit input image (visible / invisible) means switching between visible light data and invisible light data for the input image data to the recognition unit 110.

[0073] If the determination method is "landscape" and the intended use is "contrast improvement," the main control unit 117 instructs the synthesis unit 109 to perform contrast improvement processing because the contrast may be reduced due to the effects of haze, fog, etc.

[0074] If the determination method is "night view" and the intended use is "noise reduction," the main control unit 117 instructs the synthesis unit 109 to perform noise reduction processing because the amount of noise in visible light image data increases in night views.

[0075] If the determination method is "difficult light source" and the usage is "WB determination," the main control unit 117 uses invisible light image data to determine the AWB coefficients in the development unit 108, in order to prevent incorrect light source determination and unnatural color tones in the visible light image data. Specifically, it determines the light source by judging whether the visible light image data is image data of green plants such as grass or image data under mercury lamps based on the difference in reflectivity of the subject to visible and invisible light. Then, it determines the AWB coefficients according to the identified light source.

[0076] If the determination method is "increase ISO sensitivity" and the intended use is "noise reduction," then increasing the ISO sensitivity will increase the noise in the visible light image data. Therefore, the main control unit 117 causes the synthesis unit 109 to perform noise reduction processing. Alternatively, instead of the determination condition "increase ISO sensitivity," the condition "ISO sensitivity has reached its upper limit" may be used.

[0077] If the detection method is "dark and camera shake present" and the intended use is "shake correction," there is a risk that camera shake in a dark environment may cause image shake in the visible light image data. For this reason, the main control unit 117 instructs the synthesis unit 109 to perform shake correction processing. When the subject is dark, noise in the visible light image data may affect the blur kernel, so the accuracy of shake correction can be improved by estimating the blur kernel using the non-visible light image data.

[0078] When the detection method is "landscape" and the intended use is "recognition unit input image (visible / invisible) switching," it is necessary to avoid a decrease in recognition rate due to the effects of haze, fog, etc. Also, when the detection method is "night view" or "increase ISO sensitivity" and the intended use is "recognition unit input image (visible / invisible) switching," it is necessary to avoid a decrease in recognition rate due to increased noise in the visible light image data. For these reasons, in these cases, the main control unit 117 switches the input image data of the recognition unit 110 to invisible light image data. Alternatively, the recognition unit 110 may be input with both visible light image data and invisible light image data to obtain the result. When using invisible light image data, the learning model of the recognition unit 110 may be switched to a learning model for invisible light images.

[0079] Figure 5 shows an example of an LV image displayed on the display unit 121. The Ir mark 500 lights up when the use of invisible light image data is set in the UI processing of step S203, indicating to the user that invisible light image data is being used. The Ir mark 500 may be made to blink when the non-use of invisible light image data is set to prompt the user to use invisible light image data. The Ir light source mark 501 is a mark indicating that the invisible light source 122 is lit. Even if the user wants to operate the camera in a power-saving state, the use of invisible light image data and the invisible light source 122, which increase power consumption, is set to be used. By displaying marks 500 and 501 to notify the user, the user can be prompted to turn off these usage settings.

[0080] Furthermore, if invisible light image data is not used unintentionally or due to the user forgetting to enable it, the Ir Mark 500 will flash to prompt the user to use invisible light image data, thereby enabling the acquisition of better output image data.

[0081] As described above, in this embodiment, the use / non-use of invisible light image data is determined based on the imaging scene, and the driving of the invisible light image sensor 101, image processing in the second image processing unit 102, and the ON / OFF of the invisible light source 122 are controlled. This makes it possible to obtain invisible light image data, and even good output image data, while suppressing the power consumption of the camera. [Examples]

[0082] Next, Example 2 will be described. Figure 6 shows the camera configuration of Example 2. Note that components shown in Figure 6 that are the same as those shown in Example 1 (Figure 1) are denoted by the same reference numerals as in Example 1, and their descriptions are omitted.

[0083] The camera of this embodiment has an image sensor 600 instead of the visible light image sensor 100 and non-visible light image sensor 101 of Embodiment 1, and also has a non-visible light pixel separation unit (separation means) 601.

[0084] The image sensor 600 is an image sensor such as a CCD or CMOS sensor, which receives light from a subject through R, G, and B color filters and a near-infrared filter, converts the image into an electrical signal, and generates uncorrected image data.

[0085] Figure 7 shows an example of the arrangement of R, G, and B color filters and a near-infrared filter in the image sensor 600. 700 is the R color filter, 701 is the G color filter, and 703 is the B color filter. 702 is the near-infrared filter. These color filters 700, 701, 703 and the near-infrared filter 702 are arranged in the pattern shown in Figure 7 for multiple pixels on the imaging surface of the image sensor 600. The color filters 700, 701, and 703 have a peak in light transmittance in the wavelength band of each color, blocking the transmission of light in other wavelength bands. The near-infrared filter 702 also has a peak in light transmittance in the near-infrared wavelength band, blocking the transmission of light in other wavelength bands. Note that the image sensor 600 may also image invisible light in wavelength bands other than the near-infrared wavelength band.

[0086] The invisible light pixel separation unit 601 generates uncorrected visible light image data by replacing the pixels corresponding to the near-infrared filter 702 (hereinafter referred to as infrared pixels) in the uncorrected image data generated by the image sensor 600 with pixels corresponding to the G color filter 701. It then outputs the uncorrected visible light image data to the visible light image correction unit 107. The invisible light pixel separation unit 601 also generates uncorrected invisible light image data consisting only of infrared pixels in the uncorrected image data and outputs this to the invisible light image correction unit 103. In this case, the invisible light pixel separation unit 601 generates uncorrected invisible light image data by replacing all pixels corresponding to the color filters 700, 701, and 703 with infrared pixels in order to match the number of pixels in the uncorrected invisible light image data with that of the uncorrected visible light image data.

[0087] The method for separating image data in the non-visible light pixel separation unit 601 is limited to separating visible light image data from non-visible light image data, and methods other than those described above may be used. The post-separation processing of the uncorrected visible light image data and uncorrected visible light image data is the same as in Example 1.

[0088] In this embodiment as well, the use or non-use of invisible light image data is determined based on the imaging scene, and the driving of the invisible light image sensor 101, image processing in the second image processing unit 102, and the ON / OFF (light emission / non-emission) of the invisible light light source 122 are controlled accordingly. This makes it possible to obtain invisible light image data, and even good output image data, while suppressing the power consumption of the camera. [Examples]

[0089] Next, we will describe Example 3. The camera configuration in this example is the same as that of the camera in Example 1 or Example 2.

[0090] Figure 8 shows an example of an image 800 displayed on the display unit 121 of the camera in this embodiment. Image 800 may be an LV image, a still image, or a video. Image 800 shows an image obtained by capturing a night view and a person 801. In this case, the main control unit 117 determines the imaging scene based on the recognition results of the person 801 and the night view by the recognition unit 110 and the brightness information for each region from the image integration unit 111, and sets (updates) the non-visible light image usage determination information according to the determination result.

[0091] If the invisible light image usage determination information indicates the use of invisible light image data, the main control unit 117, in step 303 shown in Figure 3, turns on the drive of the infrared pixels of the invisible light image sensor 101 or image sensor 600 to enable the acquisition of invisible light image data. Furthermore, it transitions the camera to a state where it can process the invisible light image data. At this time, the main control unit 117 controls the amount of invisible light emitted from the invisible light light source 122 and the irradiation position based on the recognition result of person 801 and the brightness information of the area containing person 801, which are included in the invisible light image usage determination information.

[0092] According to this embodiment, by controlling the amount of light emitted and the irradiation position of the invisible light source 122 based on the invisible light image usage determination information, unnecessary irradiation of invisible light is prevented. As a result, the power consumption of the camera can be reduced. [Examples]

[0093] Next, we will describe Example 4. The camera configuration in this example is the same as that of Example 1 or Example 2.

[0094] Figure 9 shows an example of an image 900 output to the communication unit 123 in the camera of this embodiment. Image 900 may be an LV image, a still image, or a video. Image 900 shows an image obtained by imaging the road 901 and the vehicle 902, and is intended to record the vehicle 902 itself or the license plate number of the vehicle 902. In this case, the main control unit 117 determines the imaging scene based on the recognition result of the vehicle 902 by the recognition unit 110, the brightness information for each region from the image integration unit 111, the motion vector information from the vector detection unit 112, and the subject tracking result by the recognition unit 110. Then, according to the determination result, it sets (updates) the invisible light image usage determination information display.

[0095] If the invisible light image usage determination information indicates the use of invisible light image data, the main control unit 117, in step 303 of Figure 3, turns on the drive of the infrared pixels of the invisible light image sensor 101 or image sensor 600 to enable the acquisition of invisible light image data. Furthermore, it transitions the camera to a state where it can process the invisible light image data. At this time, the main control unit 117 predicts the position of the moving vehicle 902 in the next frame based on the recognition result of the vehicle 902 included in the invisible light image usage determination information, the brightness information of the region containing the vehicle 902, the motion vector information, and the subject tracking result. Furthermore, the main control unit 117 controls the amount of invisible light emitted from the invisible light source 122 and the irradiation position based on the predicted position of the vehicle 902 in the next frame.

[0096] In this embodiment as well, by controlling the emission amount and irradiation position of the invisible light source 122 based on the invisible light image usage determination information, unnecessary irradiation of invisible light is prevented. As a result, the power consumption of the camera can be reduced. [Examples]

[0097] Next, Example 5 will be described. The camera configuration in this example is the same as that of the camera in Example 1 or Example 2.

[0098] Figure 10 shows an example of an image 1000 displayed on the display unit 121 in the camera of this embodiment. The image 1000 may be an LV image, a still image, or a video. The image 1000 is an image obtained by capturing a person 1001, a road 1002, and a pedestrian crossing 1003. This image 1000 is acquired by an in-vehicle camera (drive recorder) and is used to record a person 1001 or a vehicle (not shown), or to detect a person 1001 crossing the pedestrian crossing 1003 for autonomous driving. The main control unit 117 determines the imaging scene based on the recognition results of the person 1001 and pedestrian crossing 1003 by the recognition unit 110, the luminance information for each region from the image integration unit 111, and the subject distance information from the distance information acquisition unit 119. Then, it sets (updates) the invisible light image usage determination information according to the determination result.

[0099] If the invisible light image usage determination information indicates the use of invisible light image data, the main control unit 117, in step 303 shown in Figure 3, turns on the drive of the infrared pixels of the invisible light image sensor 101 or image sensor 600 to enable the acquisition of invisible light image data. Furthermore, it transitions the camera to a state where it can process the invisible light image data. At this time, the main control unit 117 controls the amount of invisible light emitted from the invisible light light source 122 and the irradiation position based on the recognition results of the person 1001 and the crosswalk 1003 included in the invisible light image usage determination information, the brightness information of the area containing them, and the subject distance information.

[0100] Specifically, if the luminance information of the area containing person 1001 is lower than a predetermined luminance (i.e., person 1001 is dark) and the distance to person 1001 is greater than a predetermined distance, the amount of light emitted by the invisible light source 122 is increased. On the other hand, if the distance is closer than a predetermined distance, the amount of light emitted by the invisible light source 122 is decreased. Furthermore, if person 1001 cannot be recognized in the recognition result included in the invisible light image usage determination information, that is, if it is determined that a distant object is being imaged based only on the subject distance information, the amount of light emitted by the invisible light source 122 is increased.

[0101] In this embodiment as well, by controlling the emission amount and irradiation position of the invisible light source 122 based on the invisible light image usage determination information, unnecessary irradiation of invisible light is prevented. As a result, the power consumption of the camera can be reduced.

[0102] The above embodiment includes the following configuration.

[0103] (Configuration 1) An imaging device comprising: an imaging means that generates visible light image data and invisible light image data by imaging; a processing means that performs image processing using at least one of the visible light image data and the invisible light image data; a determination means that determines the imaging scene; and a control means that controls whether or not to perform the generation of invisible light image data by the imaging means and the image processing using the invisible light image data by the processing means according to the determination result of the imaging scene.

[0104] (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that the imaging means comprises a first imaging means for visible light and a second imaging means for non-visible light.

[0105] (Configuration 3) The imaging apparatus according to Configuration 1 or 2, characterized in that it has a separation means for separating the visible light image data and the non-visible light image data from the image data generated by the imaging means.

[0106] (Configuration 4) The imaging device according to any one of Configurations 1 to 3, characterized in that the determination means determines the imaging scene according to the settings made by the user.

[0107] (Configuration 5) The imaging device according to any one of Configurations 1 to 4, characterized in that the determination means determines the imaging scene according to the result of subject recognition using at least one of the visible light image data and the non-visible light image data.

[0108] (Configuration 6) The imaging apparatus according to Configuration 5, characterized in that the determination means selects image data from the visible light image data and the non-visible light image data to be used for subject recognition according to the imaging scene.

[0109] (Configuration 7) The imaging apparatus according to Configuration 5 or 6, wherein the determination means uses a learning model for subject recognition, and when the image data used for subject recognition is visible light image data and when it is non-visible light image data, respectively, it uses a learning model for visible light images and a learning model for non-visible light images as the learning model.

[0110] (Configuration 8) The imaging apparatus according to any one of Configurations 1 to 7, characterized in that the determination means determines the imaging scene using brightness information obtained from at least one of the visible light image data and the non-visible light image data.

[0111] (Configuration 9) The imaging apparatus according to any one of Configurations 1 to 8, characterized in that the determination means determines the imaging scene using motion vector information obtained from at least one of the visible light image data and the non-visible light image data.

[0112] (Configuration 10) The imaging device according to any one of Configurations 1 to 9, characterized in that the determination means determines the imaging scene using information on the distance to the subject.

[0113] (Configuration 11) The imaging device according to any one of Configurations 1 to 10, characterized in that the control means controls whether or not to generate the invisible light image data and perform image processing using the invisible light image data, according to the determination result of the imaging scene and the detection result of the shaking of the imaging device.

[0114] (Configuration 12) The imaging apparatus according to any one of Configurations 1 to 11, characterized in that the control means controls whether or not to emit an invisible light source that irradiates the subject with invisible light, according to the determination result of the imaging scene.

[0115] (Configuration 13) When the generation of non-visible light image data and image processing using the non-visible light image data are performed, the control means records the non-visible light image data in memory according to user settings, as described in any one of Configurations 1 to 12.

[0116] (Composition 14) The imaging apparatus according to any one of configurations 1 to 13, characterized by having a display means for displaying to the user that the generation of non-visible light image data and image processing using the invisible light image data are being performed.

[0117] (Composition 15) The imaging apparatus according to any one of configurations 1 to 14, characterized in that it has a display means that prompts the user to perform the generation of non-visible light image data and image processing using the said non-visible light image data when these processes are not performed.

[0118] (Configuration 16) An imaging device according to any one of Configurations 1 to 15, characterized in that it has a display means that displays to the user that an invisible light source that irradiates an object with invisible light is lit. (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0119] The embodiments described above are merely representative examples, and various modifications and changes can be made to each embodiment when implementing the present invention. [Explanation of Symbols]

[0120] 100 Visible light image sensor 101 Non-visible light image sensor 102 Second Image Processing Unit 106 First Image Processing Unit 110 Recognition part 111 Image Integration Section 112 Vector detection unit 117 Main Control Unit 118 Camera status acquisition unit 119 Distance information acquisition section

Claims

1. A control means for acquiring visible light image data and non-visible light image data by imaging means; a processing means for performing image processing using at least one of the visible light image data and the non-visible light image data; A determination means for determining an imaging scene, The control means controls whether or not to perform imaging for the non-visible light image data by the imaging means and image processing using the non-visible light image data in the processing means, depending on the result of the determination of the imaging scene.

2. 2. The imaging control device according to claim 1, wherein the imaging means comprises a first imaging means for visible light and a second imaging means for non-visible light.

3. 2. The imaging control device according to claim 1, further comprising a separation unit that separates the visible light image data and the non-visible light image data from the image data generated by the imaging unit.

4. 2. The imaging control device according to claim 1, wherein the determination means determines the imaging scene in accordance with a setting made by a user.

5. 2. The imaging control device according to claim 1, wherein the determination means determines the imaging scene according to a result of subject recognition using at least one of the visible light image data and the non-visible light image data.

6. 6. The imaging control device according to claim 5, wherein the determination means selects the image data to be used for the subject recognition from among the visible light image data and the non-visible light image data in accordance with the imaging scene.

7. The determination means is A learning model is used for the object recognition, The imaging control device according to claim 5, characterized in that when the image data used for the subject recognition is the visible light image data and when the image data is the non-visible light image data, a learning model for visible light images and a learning model for non-visible light images are used as the learning models, respectively.

8. 2. The imaging control device according to claim 1, wherein the determination means determines the imaging scene using luminance information obtained from at least one of the visible light image data and the non-visible light image data.

9. 2 . The imaging control device according to claim 1 , wherein the determination means determines the imaging scene using information on a motion vector obtained from at least one of the visible light image data and the non-visible light image data.

10. 2. The imaging control device according to claim 1, wherein the determination means determines the imaging scene using information on a subject distance.

11. The imaging control device according to claim 1, characterized in that the control means controls whether or not to generate the non-visible light image data and to execute image processing using the non-visible light image data, depending on the result of determining the imaging scene and the result of detecting shake of the imaging device.

12. 2. The imaging control device according to claim 1, wherein the control means controls whether or not to emit an invisible light source for irradiating an object with invisible light, depending on the result of the determination of the imaging scene.

13. 2. The imaging control device according to claim 1, wherein when the generation of the non-visible light image data and image processing using the non-visible light image data are performed, the control means records the non-visible light image data in a memory in accordance with a setting made by a user.

14. An imaging means; An imaging device comprising the imaging control device according to claim 1.

15. 15. The imaging apparatus according to claim 14, further comprising a display unit that displays to a user that the non-visible light image data is being generated and image processing using the non-visible light image data is being executed.

16. 15. The imaging device according to claim 14, further comprising a display unit that displays a message urging a user to generate the non-visible light image data and to perform image processing using the non-visible light image data when the generation of the non-visible light image data and the image processing using the non-visible light image data have not been performed.

17. 15. The imaging apparatus according to claim 14, further comprising a display unit that displays to a user that a light source of invisible light that irradiates an object with invisible light is turned on.

18. A control method for an imaging control device having a control means for acquiring visible light image data and non-visible light image data by imaging with an imaging means, and a processing means for performing image processing using at least one of the visible light image data and the non-visible light image data, determining an imaging scene; The control method is characterized in that the control means controls whether or not to perform imaging for the non-visible light image data by the imaging means and image processing using the non-visible light image data in the processing means, depending on the result of the determination of the imaging scene.

19. 20. A program for causing a computer of an imaging device to execute a process according to the control method of claim 18.

Images