Imaging device, its control method, and program

Abstract

The present invention provides an imaging device, a control method thereof, and a program that can perform optimal imaging control for each pixel in each region of the image sensor while minimizing the time lag until image capture. [Solution] The imaging device 100, which includes an image sensor unit having multiple pixels, has an image sensor control unit 103 containing a feature region detection unit 102 that detects a feature region based on pixel information obtained from the multiple pixels, and an image sensor control unit 103 that causes pixels in the feature region to be photographed under a second shooting condition different from a first shooting condition set according to the user's instructions, and causes pixels in other regions other than the feature region to be photographed under the first shooting condition.

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method thereof, and a program, and particularly to an imaging device that switches the control of pixels in a pixel element unit for each region, a control method thereof, and a program.

Background Art

[0002] It is known that when photographing light that rapidly turns on and off, such as an LED, with an exposure time faster than the on-off cycle, unevenness in the on-off (hereinafter referred to as flicker) occurs in the photographed image.

[0003] Particularly in recent years, due to the widespread use of digital signage and the like, when digital signage appears as part of an image, only a partial region of the image may be affected by flicker.

[0004] In order to suppress the influence of flicker, it is known to take a photograph with the exposure time adjusted to an integer multiple of the flicker frequency. However, with this method, the user cannot take a photograph with an arbitrary exposure time, and subject blur may occur when photographing a subject that is moving violently.

[0005] As described above, in a scene where digital signage appears in the angle of view, the exposure time optimal for flicker suppression and the exposure time optimal for subject blur suppression may be different.

[0006] For example, in Patent Document 1, in HDR photography, a method is proposed in which pixels arranged in an array in a pixel element unit are divided into short storage regions and long storage regions, and the exposure time is controlled for each region.

Prior Art Documents

Patent Documents

[0007]

Patent Document 1

Summary of the Invention

[0008] However, the prior art disclosed in Patent Document 1 above requires time to divide the region into a short-exposure region and a long-exposure region, and to control the exposure time for each divided region, resulting in a time lag before shooting.

[0009] Therefore, in scenes where the subject is moving or moves rapidly within a short period of time, this time lag makes it difficult to take a picture at the desired moment.

[0010] Therefore, in view of the above, the object of the present invention is to provide an imaging device, a control method thereof, and a program that can perform optimal imaging control for each pixel in each region of the image sensor while suppressing the time lag until imaging. [Means for solving the problem]

[0011] To solve the above problems, the imaging device according to claim 1 of the present invention is an imaging device comprising an image sensor unit having a plurality of pixels and a setting unit that sets first shooting conditions according to user instructions, wherein the imaging device has inside the image sensor unit: detection means for detecting a feature region based on pixel information obtained from the plurality of pixels, and control means for causing the pixels of the feature region to be photographed under a second shooting condition different from the first shooting condition, and the pixels of other regions other than the feature region to be photographed under the first shooting condition. [Effects of the Invention]

[0012] According to the present invention, it is possible to perform optimal shooting control for each pixel in each region of the image sensor while suppressing the time lag until shooting. [Brief explanation of the drawing]

[0013] [Figure 1] This is a block diagram showing the hardware configuration of an imaging device according to the first embodiment. [Figure 2] This is a timing chart for acquiring pixel information for feature region detection, detecting the feature region, and controlling the shooting process according to the first embodiment. [Figure 3] This figure shows an example of an image captured when exposure time is controlled for each region. [Figure 4] This is a flowchart of the flickerless imaging process according to the first embodiment. [Figure 5] This figure shows the method for setting shooting conditions for each region according to the first embodiment. [Figure 6] This is a diagram illustrating HDR shooting according to the second embodiment. [Figure 7] This is a flowchart of the HDR shooting process according to the second embodiment. [Figure 8] This figure illustrates region-specific AF control using multiple photodiodes according to a third embodiment. [Figure 9] This is a flowchart of the AF control process according to the third embodiment. [Figure 10] This figure illustrates the low-resolution imaging of a non-focus area according to the fourth embodiment. [Modes for carrying out the invention]

[0014] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0015] (First embodiment) Figure 1 is a block diagram showing an example of the schematic configuration of the imaging device 100 according to the first embodiment. The imaging device 100 according to this embodiment is, for example, an interchangeable-lens digital camera, which can be fitted with any lens and has a still image capturing function and a moving image capturing function.

[0016] In FIG. 1, the imaging device 100 includes an imaging element unit 104, a system control unit 105, a recording unit 106, a display unit 107, and an operation unit 108.

[0017] The imaging element unit 104 is a CMOS image sensor composed of a pixel unit 101, a feature region detection unit 102, and an imaging element control unit 103.

[0018] The pixel unit 101 is composed of a plurality of pixels. For the optical image of the subject, each pixel performs photoelectric conversion to generate charges corresponding to the incident light amount, converts them into respective electrical signals, generates digital image data, and outputs it. Note that the image data generated here is composed of pixel information from each pixel. At this time, the pixel unit 101 can also amplify the electrical signal by performing gain control. In addition, the pixel unit 101 also has an electronic shutter function for adjusting the incident light amount to each pixel and can control the exposure time.

[0019] The feature region detection unit 102 (detection means) detects a feature region based on the pixel information obtained from the pixel unit 101. The feature region means a characteristic region in the image region of the image data generated by the pixel unit 101, and examples include a flicker region and a subject region.

[0020] The imaging element control unit 103 controls the gain control and exposure time control of the pixel unit 101. It also controls the readout timing and method of the pixel information from the pixel unit 101.

[0021] Note that the imaging element control unit 103 can control the exposure time control, gain control, and readout control of the pixel information of the pixel unit 101 for each region based on the detection result of the feature region detection unit 102.

[0022] In addition, the imaging element control unit 103 outputs the pixel information obtained from the pixel unit 101 to the system control unit 105.

[0023] The system control unit 105 is responsible for controlling the entire imaging device 100. It also contains a CPU, RAM, and ROM (none of which are shown), and the CPU loads the program in ROM into RAM and reads it sequentially to execute the high-frequency flickerless imaging process described later, but the entity that executes this process will henceforth be simply referred to as the system control unit 105.

[0024] The recording unit 106 records image data and other data obtained from the image sensor unit 104.

[0025] The display unit 107 displays image data obtained from the image sensor unit 104 and menus.

[0026] The control unit 108 is used by the user to set shooting conditions, etc.

[0027] Next, using the timing chart in Figure 2, the acquisition of pixel information for feature region detection, feature region detection, and shooting control according to this embodiment will be explained.

[0028] This section describes a case where pixel information for feature region detection is acquired during the accumulation of the Nth frame.

[0029] Figure 2(a) shows an example of control timing when the image sensor unit 104 has a feature region detection unit 102 and an image sensor control unit 103 inside. In this control case, since the feature region detection unit 102 inside the image sensor unit 104 performs feature region detection, there is no need to transfer pixel information to a feature region detection unit outside the image sensor unit 104. Therefore, feature region detection 202a can be performed immediately using the pixel information acquired in the Nth frame's accumulation 201a. In addition, in this control case, the image sensor control unit 103 inside the image sensor unit 104 performs shooting control 203a on the pixel unit 101 of the feature region detected in feature region detection 202a. Therefore, there is no need to transfer feature region information to an image sensor control unit outside the image sensor unit 104, and control can be performed that immediately reflects the feature region detection result based on the Nth frame's accumulation 201a in the N+1th frame's accumulation 204a.

[0030] Figure 2(b) shows an example of control timing when the system control unit 105, which is outside the image sensor unit 104, has a feature region detection unit 102 and an image sensor control unit 103. In this control case, time is required for reading out the pixel information acquired in the Nth frame's accumulation 201b, and time is required to transfer the pixel information to the system control unit 105. Therefore, the time lag T21 from accumulation 201b to feature region detection 202b is larger than the time lag T11 from accumulation 201a to feature region detection 202a. Also, the difference between the time lag T11 and the time lag T21 increases as the total number of pixels in the pixel unit 101 increases. Furthermore, when the system control unit 105 controls the image sensor unit 104, constraints such as periodic shooting control dependent on the frame rate arise. Therefore, the time lag T22 between the detection of a feature region and the start of shooting control 203a inside the image sensor unit 104 is greater than the time lag T12 between the detection of a feature region and the start of shooting control 203a inside the image sensor unit 104.

[0031] The above describes the case where feature region detection 202a and shooting control 203a are performed inside the image sensor unit 104 as shown in Figure 2(a), and the case where feature region detection 202b and shooting control 203b are performed outside the image sensor unit 104 as shown in Figure 2(b). In the case of Figure 2(b), the time lag is larger from the acquisition of pixel information for feature region detection in the accumulation of the Nth frame to the detection of the feature region and the actual execution of shooting control for the detected feature region compared to the case of Figure 2(a). Therefore, in the case of Figure 2(b), there is not enough time to reflect the detection result of the Nth frame in the accumulation 204b of the N+1th frame, and it is reflected in the accumulation 205b of the N+2nd frame.

[0032] As described above, in this embodiment, by providing the feature region detection unit 102 and the image sensor control unit 103 inside the image sensor unit 104, pixel information acquisition, feature region detection, and region-specific shooting control can be performed immediately. Therefore, the time lag from region detection to shooting control can be reduced compared to the case shown in Figure 2(b). On the other hand, since the processing load of the image sensor unit 104 will increase, there is a possibility that the heat generation of the image sensor unit 104, which has been a problem in recent years, will further increase. For this reason, a second feature region detection unit may be provided outside the image sensor unit 104, for example, in the system control unit 105. This makes it possible to switch the detection of feature regions from the feature region detection unit 102 to the second feature region detection unit depending on the subject and the state of the image sensor unit 104, thereby suppressing the increase in heat generation of the image sensor unit 104.

[0033] Next, Figure 3 shows an example of an image captured when exposure time is controlled for each region. In this embodiment, flickerless imaging will be described.

[0034] The flicker regions R301, R303, and R305 represent the areas within the pixel region 101 where flicker occurs in each captured image shown in Figures 3(a) to (c) (flicker regions: characteristic regions). On the other hand, the normal regions R302, R304, and R306 represent the areas within the pixel region 101 where flicker does not occur in each captured image (areas other than characteristic regions). The diagram on the right of each captured image shows the exposure time for each area of ​​the pixel region 101. Below, the flicker regions R301, R303, and R305 will be described as characteristic regions.

[0035] Here, we will explain an example where the flicker regions R301, R303, and R305 show an LED signage display that flashes repeatedly at 500Hz, while the normal regions R302, R304, and R306 show a scene with a moving subject.

[0036] Figure 3(a) shows an image of a captured image taken with an exposure time t31 (=1 / 500 sec), which is an integer multiple of the reciprocal of the flicker frequency, in order to reduce the effect of flicker in both the flicker region R301 and the normal region R302. While this exposure time can reduce the effect of flicker, the subject moves during the exposure period, resulting in blurring of the subject in the normal region R302, as shown in Figure 3(a).

[0037] Figure 3(b) shows an image of a captured image taken with a short exposure time t32 (=1 / 2000 sec) in both the flicker region R301 and the normal region R302, in order to prevent blurring of the subject. When taken with this exposure time, as shown in Figure 3(b), no blurring occurs in the subject within the normal region R304, but because the exposure time is shorter than the reciprocal of the flicker frequency, the image of the LED signage in the flicker region R303 is affected by flicker.

[0038] Figure 3(c) shows an image of the captured image when the flicker region R305 is photographed with an exposure time of t21 to reduce the effect of flicker, and the normal region R306 is photographed with an exposure time of t32 to prevent subject blurring. By photographing the flicker region R305 and the normal region R306 with different exposure times suitable for each region, as shown in Figure 3(b), subject blurring does not occur in the normal region R306, and the effect of flicker can be reduced in the flicker region R305.

[0039] In the above explanation, an example was shown in which the flicker region is used as a feature region and the exposure time is controlled for the pixel portion 101. However, the control method for the feature region and the pixel portion 101 is not limited to this.

[0040] Next, the high-frequency flickerless imaging process according to the first embodiment will be explained using the flowchart in Figure 4.

[0041] In step S401, the system control unit 105 (setting unit) sets the shooting conditions (first shooting conditions) for the image sensor unit 104 according to the user's instructions. The shooting conditions set here include, for example, the exposure time in the normal range (first exposure time) and the frame rate, using values ​​set by the user using the operation unit 108.

[0042] In step S402, when the system control unit 105 detects a shooting start command from the user, who is the photographer, using the operation unit 108, it issues an imaging command to the image sensor control unit 103. Upon receiving this imaging command, the image sensor control unit 103 controls the pixel unit 101 and the feature area detection unit 102 to acquire pixel information for feature area detection from the pixel unit 101. The pixel information for feature area detection acquired here is not particularly limited, but examples include the live view display image or still image of the previous frame.

[0043] In step S403, the system control unit 105 controls the feature region detection unit 102 to detect flicker regions based on the pixel information for feature region detection acquired in step S402. At this time, the feature region detection unit 102 detects flicker regions using a trained model based on training data, for example. Specifically, the trained model detects the installation area of ​​the signage from the shape of the signage identified based on edge information, etc., and determines that the detected installation area is a flicker region. However, the flicker region detection means is not limited to this. For example, the way in which flicker fringes occur can be estimated from the frequency information of general flicker and the readout speed and exposure time of the image data in the pixel unit 101. Therefore, flicker regions may be detected from the estimation result and the amount of brightness change in the image for feature region detection acquired in step S402. After that, the feature region detection unit 102 transmits the detected flicker regions to the image sensor control unit 103.

[0044] In step S404, the system control unit 105 controls the image sensor control unit 103 to perform a process for determining shooting conditions for each region. In this process, first, the image sensor control unit 103 (control means) detects the flicker frequency (feature quantity) in the flicker region detected in step S403. Then, based on the detected flicker frequency and the shooting settings set in step S401, the image sensor control unit 103 determines the shooting conditions (second shooting conditions) for exposure time settings such as the accumulation time of the flicker region, exposure start timing, and readout timing. Details of the process in step S404 will be described later with reference to Figure 5.

[0045] In step S405, the system control unit 105 controls the image sensor control unit 103 to perform shooting control for the pixels in the flicker region detected in step S403 in the pixel unit 101. Through this shooting control, the image sensor control unit 103 (control means) causes the pixels in the flicker region to be photographed under the shooting conditions determined in step S404. After the pixel information generated by the pixels in the flicker region is accumulated, the process proceeds to step S407.

[0046] In step S406, the system control unit 105 controls the image sensor control unit 103 to perform imaging control for pixels in the normal region of the pixel unit 101. Through this imaging control, the image sensor control unit 103 (control means) causes pixels in the normal region to be imaged under the normal region imaging conditions determined in step S404. After the pixel information generated by the pixels in the normal region is accumulated, the process proceeds to step S407.

[0047] In step S407, the system control unit 105 controls the image sensor control unit 103 to read the pixel information accumulated in steps S405 and S406 from the pixel unit 101. The image sensor control unit 103 outputs the pixel information read from the pixel unit 101 to the system control unit 105 as a single image data file, controlled under shooting conditions suitable for the flicker region and the normal region, respectively. The system control unit 105 performs image processing on the image data received from the image sensor control unit 103 to record it to the recording unit 106 or display it on the display unit 107. This image processing also includes generating video data using the image data received from the image sensor control unit 103 as frame images. After that, this process ends.

[0048] Next, using Figure 5, the process by the image sensor control unit 103 for determining shooting conditions for each region according to the first embodiment will be explained. Here, an example is shown for shooting an LED signage with a flicker frequency of XHz. t51 represents the exposure time of pixels in the flicker region (second exposure time), and t52 represents the exposure time of pixels in the normal region (first exposure time). In the case of normal shooting where the exposure time of all pixels in the pixel unit 101 is a uniform exposure time (t51==t52), T1 is the time when exposure can start, T2 is the time when reading starts, and t53 is the time from T1 to T2 (=exposure time).

[0049] First, the reciprocal of the flicker frequency XHz, 1 / X, is compared with the exposure time t53. If the reciprocal of the flicker frequency, 1 / X, is less than or equal to the exposure time t53, the readout start time remains unchanged at T2, and the exposure time t51 of the flicker region is set to n × 1 / X ​​(where n is an integer). In this case, n is the integer n that minimizes the difference between n × 1 / X ​​and t52, among integers n such that n × 1 / X ​​does not exceed t53. In this case, the exposure time t52 of the normal region is, for example, the exposure time tu set by the user in step S401.

[0050] If the reciprocal of the flicker frequency XHz, 1 / X, is longer than the exposure time t53, the exposure time t51 of the flicker region is set to 1 / X, exposure of the flicker region is started from T1, the time when exposure can be started, and the readout start time is changed to T3, the time when the exposure is finished. When changing the readout start time from T2 to a later time in this way, for example, the system control unit 105 may control the imaging device 100 to reduce the frame rate.

[0051] Furthermore, regarding the exposure start timing for the normal region and the flicker region, exposure is controlled to align the exposure centroids of each region in order to reduce the sense of incongruity with the pixel information of the normal region and the flicker region. Here, the exposure centroid refers to the median time from the start to the end of exposure for all pixels within each region.

[0052] In this embodiment, a feature region is detected in the pixel section 101, and different shooting controls are performed for the detected feature region and the other regions (=normal regions). As a result, even in scenes where the subject is moving quickly, a single image can be captured without blurring of the subject and with reduced flicker effects.

[0053] Furthermore, the example of determining the shooting conditions for each region shown in Figure 5 is just one example in this embodiment, and the method for determining the shooting conditions for each region is not limited to this. For example, the flicker frequency may be detected in advance, and when setting the shooting conditions in step S401, the exposure time tf of the flicker region may be set in advance, and the exposure time t51 of the flicker region detected in step S403 may be set as the exposure time tf.

[0054] Furthermore, while the example of determining shooting conditions for each region shown in Figure 5 uses a method that controls the exposure time of the flicker region, this method is not limited to any method that can determine shooting conditions that reduce the effect of flicker. For example, the effect of flicker may be reduced by setting shooting conditions that accumulate multiple times only in the flicker region and obtaining the average of the multiple pixel information obtained.

[0055] (Second embodiment) Next, we will explain how to determine the shooting conditions for each region of the pixel section 101 in the second embodiment.

[0056] In the first embodiment, a method for determining the shooting conditions for each region when performing flickerless shooting was described, whereas in this embodiment, a method for determining the shooting conditions for each region when performing HDR shooting will be described.

[0057] In this embodiment, hardware components similar to those in the first embodiment are denoted by the same reference numerals, and redundant explanations are omitted.

[0058] First, we will explain HDR photography using Figure 6.

[0059] Low-luminance regions R601 and R603 represent the low-luminance areas of each captured image shown in Figures 6(a) and (b), while normal regions R602 and R604 represent the areas of each captured image other than the low-luminance regions. The low-luminance regions will be described below as characteristic regions.

[0060] Figure 6(a) shows an image of a scene captured with a gain setting suitable for the normal region R602. In scenes with strong sunlight, as shown in Figure 6, the difference in brightness between the low-luminance region R601 and the normal region R602 becomes large. With a gain setting suitable for the normal region R602, the low-luminance region R601 becomes completely black. Conversely, with a gain setting suitable for the low-luminance region R601, the normal region R602 becomes overexposed.

[0061] Figure 6(b) shows an image of the scene captured in the same way as in Figure 6(a), but with the gain of the normal region R602 remaining unchanged, and only the gain of the low-luminance region R603 increased.

[0062] In this way, by increasing the gain only in the low-luminance region R603, the brightness of the low-luminance region R603 can be changed without changing the exposure conditions of the normal region R604. Therefore, it is possible to take pictures with exposure conditions suitable for both the low-luminance region R603 and the normal region R604. However, if a time lag occurs between the detection of the low-luminance region and the gain control for each region, a discrepancy will occur between the actual low-luminance region and the region where the gain is increased due to the angle of view or movement of the subject, resulting in an increase in the gain of the normal region. Such an increase in the gain of the normal region leads to the problem of an unnaturally bright image appearing in a part of the image. In this embodiment, since the image sensor unit 104 has a feature region detection unit 102 and an image sensor control unit 103, the time lag from detection to shooting control can be reduced, thus suppressing the unnatural brightness of a part of the image.

[0063] Next, the HDR shooting process according to the second embodiment will be explained using the flowchart in Figure 7.

[0064] In step S701, similar to step S401, the system control unit 105 sets the shooting conditions for the image sensor unit 104 according to the user's instructions. The shooting conditions set here are, for example, the gain setting in the normal range, using a value set by the user using the operation unit 108.

[0065] In step S702, similar to step S402, when the system control unit 105 detects a shooting start instruction from the user, who is the photographer, using the operation unit 108, it issues an imaging instruction to the image sensor control unit 103. Upon receiving this imaging instruction, the image sensor control unit 103 controls the pixel unit 101 and the feature area detection unit 102 to acquire pixel information for feature area detection from the pixel unit 101.

[0066] In step S703, the system control unit 102 controls the feature region detection unit 102 to calculate a luminance distribution (feature quantity) from the pixel information for feature region detection acquired in step S702, and to detect areas with luminance lower than a threshold as low-luminance regions. Subsequently, the feature region detection unit 102 transmits the detected low-luminance regions and luminance distribution to the image sensor control unit 103.

[0067] In step S704, the system control unit 105 controls the image sensor control unit 103 to perform a process for determining the shooting conditions for each region. In this process, a gain is set for the pixel portion 101 of the low-luminance region detected in step S703. The gain set at this time is determined based on the luminance information detected in step S703 so that the difference between the average luminance of the low-luminance region and the average luminance of the normal region is within a threshold.

[0068] In step S705, the system control unit 105 controls the image sensor control unit 103 to perform different shooting controls for the low-luminance region and the normal region of the pixel unit 101. Through this shooting control, the image sensor control unit 103 controls the pixel unit 101 to accumulate optical signals, converts them into electrical signals by applying different gains for each region set in steps S701 and S704, and generates pixel information. The pixel information generated by this shooting control is accumulated in the pixel unit 101.

[0069] In step S706, similar to step S407, the system control unit 105 controls the image sensor control unit 103 to read the pixel information accumulated in step S705 from the pixel unit 101. The image sensor control unit 103 outputs the pixel information read from the pixel unit 101 as a single image file to the system control unit 105. The system control unit 105 performs image processing on the image data received from the image sensor control unit 103 to record it to the recording unit 106 or display it on the display unit 107. This image processing also includes generating video data using the image data received from the image sensor control unit 103 as frame images. After that, this process is terminated.

[0070] Thus, in this embodiment, HDR shooting can be achieved by capturing a single image.

[0071] Note that the determination of gain for each region shown in Figure 7 is just one example in this embodiment, and is not limited to this method as long as HDR shooting is possible. For example, exposure control methods such as exposure time and ISO sensitivity settings may be used.

[0072] (Third embodiment) Next, the determination of the shooting conditions for each region of the pixel section 101a in the third embodiment will be described.

[0073] In the first embodiment, the method for determining the shooting conditions for each region when performing flickerless shooting was described, and in the second embodiment, the method for determining the shooting conditions for each region when performing HDR shooting was described. In contrast, this embodiment describes the method for determining the shooting conditions (readout method settings) for each region when performing AF (autofocus) control.

[0074] In the first and second embodiments, a pixel section 101 was used in which one photodiode was provided within one pixel. In contrast, in this embodiment, a pixel section 101a is used in which multiple photodiodes capable of independently reading pixel information are provided within one pixel.

[0075] In this embodiment, hardware components similar to those in the first embodiment are denoted by the same reference numerals, and redundant descriptions are omitted.

[0076] First, using Figure 8, we will explain the region-specific autofocus (AF) control using multiple photodiodes according to this embodiment. In the example shown in Figure 8, each pixel constituting the pixel section 101a is provided with four photodiodes 801a to 801d.

[0077] In this embodiment, as an example of a method for calculating the focus position, a distance calculation means that compares the phase difference of the optical signals of multiple photodiodes within a single pixel will be described. Here, the image data obtained from the photodiodes 801a and 801b on the left side of each pixel is called the left image, and the image data obtained from the photodiodes 801c and 801d on the right side of each pixel is called the right image. In the example in Figure 8, each pixel of the pixel section 101 has four photodiodes, but this embodiment is not limited to this, and each pixel may have two or more photodiodes.

[0078] The normal area R802 represents the area that does not include the subject targeted for AF, while the subject area R803 represents the area that includes the subject targeted for AF. Below, the subject area R803 will be described as the feature area.

[0079] The pixels indicated by the shaded area in Figure 8 represent pixels located in the subject region R803. In such scenes, the focus position calculation for focusing on the subject is performed using the pixels indicated by the shaded area, so pixels in the normal region R802 are not needed for the focus position calculation. Therefore, only the pixels in the subject region R803 are read from the right and left images respectively and used for the focus position calculation. At this time, if there is a time lag between the detection of the subject region and the readout control, the movement of the subject may cause pixels in areas without a subject to be used for the focus position calculation, negatively affecting the processing time and the accuracy of the focus position calculation. In this embodiment, the time lag from the detection of the subject region to the readout can be reduced, so even in scenes where the subject moves quickly, the readout of pixels in the subject region can be accurately controlled, and the negative impact on processing time and the accuracy of the focus position calculation can be reduced.

[0080] This eliminates the need to separate the reading of pixels that do not require focus position calculation into left and right images, thereby reducing power consumption and speeding up the processing time for focus position calculation.

[0081] Next, the AF control process according to the third embodiment will be explained using the flowchart in Figure 9.

[0082] In step S901, similar to step S401, the system control unit 105 sets the shooting conditions for the image sensor unit 104 according to the user's instructions. In addition, it obtains the user's designation of the subject to be detected, such as "person," from the operation unit 108.

[0083] In step S902, similar to step S402, when the system control unit 105 detects a shooting start instruction from the user, who is the photographer, using the operation unit 108, it issues an imaging instruction to the image sensor control unit 103. Upon receiving this imaging instruction, the image sensor control unit 103 controls the pixel unit 101a and the feature area detection unit 102 to acquire pixel information for feature area detection from the pixel unit 101a.

[0084] In step S903, the system control unit 105 controls the feature region detection unit 102 and detects the subject region in the pixel unit 101a based on the pixel information for feature region detection acquired in step S902. Specifically, it detects the subject region in which the relevant subject (for example, an object classified as "person": feature quantity) exists, based on a trained model that has been previously trained based on training data. The trained model here refers to a model that has been trained to classify objects that appear in each region from the feature regions in the image, and if there is a relevant subject, it outputs that region as the detection result. However, the subject region detection means is not limited to this. For example, the subject region R803 may be specified in response to a user's touch operation on the touch panel of the display unit 107 during live view display. However, the method is not limited to this embodiment as long as the subject region R803 can be identified by some means. For example, the subject region R803 may be specified in response to a user's drag operation on the AF frame displayed on the touch panel.

[0085] In step S904, the system control unit 105 controls the image sensor control unit 103 and controls the exposure of the pixel unit 101a based on the shooting conditions set in step S901, and the pixel unit 101a accumulates the right image and the left image for each pixel.

[0086] In step S905, the system control unit 105 controls the image sensor control unit 103 to independently read out the right image and left image accumulated in step S904 from each pixel in the subject area detected in step S903. Subsequently, the image sensor control unit 103 outputs the right image and left image read out from each pixel in the subject area as pixel information to the system control unit 105. Then, the process proceeds to step S907.

[0087] In step S906, the system control unit 105 controls the image sensor control unit 103 to perform additive synthesis of the right image and the left image accumulated in step S904 for each pixel in the area outside the subject area (normal area) of the pixel unit 101a. Subsequently, the image sensor control unit 103 reads the pixel information obtained from the additive synthesis from each pixel in the normal area and outputs it to the system control unit 105. At this time, the pixel information output in step S906 is used for purposes such as development processing, which is carried out together with the pixel information acquired in step S905 and displayed on the display unit 107, but it is not used for focus position calculation. Therefore, depending on the shooting settings, such as when an image for focus position calculation is not displayed, step S906 does not need to be performed. After that, the process proceeds to step S907.

[0088] In step S907, the system control unit 105 compares the phase difference between the right image and the left image acquired in step S905, calculates the focus position, and then terminates this process.

[0089] In this embodiment, the focus position calculation is performed only for the subject area and not for the normal area. Therefore, power consumption can be reduced and the processing time for the focus position calculation can be sped up.

[0090] Although the phase difference method was used in the above-described focus position calculation, this is just one example, and the method of calculating the focus position is not limited to this method; for example, a contrast method may also be used.

[0091] (Fourth embodiment) Next, we will explain how to determine the shooting conditions for each region of the pixel section 101 in the fourth embodiment.

[0092] In the third embodiment, AF control was described using the subject area, which is the area of ​​interest for the user (hereinafter referred to as the area of ​​interest), as the feature area. In this embodiment, however, low-resolution shooting is described using a non-interest area (normal area) other than the subject area as the feature area.

[0093] In this embodiment, hardware components similar to those in the first embodiment are denoted by the same reference numerals, and redundant explanations are omitted.

[0094] First, using Figure 10, we will explain the low-resolution imaging of non-focus areas according to this embodiment.

[0095] In fields such as VR (Virtual Reality), a known method for reducing processing load is to render the area of ​​interest at high resolution and the non-interesting area at a lower resolution.

[0096] Similarly, in image capture, for example, for display images, high resolution is required in the subject area, which is important for determining composition and confirming focus. However, by lowering the resolution in the normal area, data volume and power consumption can be reduced. Therefore, by reading only the subject area at high resolution and downsampling the normal area, the resolution can be reduced, resulting in reduced data volume and power consumption.

[0097] In Figure 10, the subject area R1001 represents the subject area that is rendered at high resolution, while the normal area R1002 represents the area other than the subject area R1001 that is rendered at a lower resolution. The subject area R1001 is also represented by the shaded area in Figure 10.

[0098] The subject area R1001 is an important area for the user when taking a picture, so no decimation is performed and all pixels are read out. On the other hand, in the normal area R1002, pixels are read out with decimation to reduce the amount of data and power consumption. Figure 10 shows an example where only the normal area R1002 is read out with 1 / 2 pixels decimated.

[0099] In this embodiment, low-resolution imaging of non-target areas can be achieved by changing the downsampling rate depending on the region. The control of low-resolution imaging of non-target areas according to this embodiment will be explained below using the flowchart in Figure 9.

[0100] Steps S901 to S903 are the same as those described in the third embodiment, so redundant explanations will be omitted. In step S903, the area of ​​the pixel section 101 other than the detected subject area is set as the normal area R1002 described above.

[0101] In step S904, the system control unit 105 controls the image sensor control unit 103 and controls the exposure of the pixel unit 101 based on the shooting conditions set in step S901. Pixel information generated by this exposure control is stored in the pixel unit 101.

[0102] In step S905, the system control unit 105 controls the image sensor control unit 103 to read out all the pixel information accumulated at each pixel in the subject area detected in step S903. Then, the process proceeds to step S907.

[0103] In step S906, the system control unit 105 controls the image sensor control unit 103 to read out pixel information by thinning out pixels from the normal area R1002 of the pixel unit 101 at a predetermined thinning rate (here, 1 / 2 in the vertical direction). Subsequently, the system control unit 105 performs development processing, etc., based on the pixel information received from the image sensor control unit 103. At this time, since it is necessary to adjust the image size for display, the pixel information of two rows in the vertical direction of the subject area R1002 read out in step S905 is added and averaged to adjust the image size and output as a single image. This makes it possible to obtain an image with high resolution only for the feature area. However, the method of adjusting the image size is not limited to this method; it may also be changed depending on the shooting settings and shooting environment, such as adjusting the image size by linearly interpolating the thinned pixels of the non-focus area R1001.

[0104] Thus, in this embodiment, it is possible to perform imaging that reduces the amount of data and saves power without reducing the resolution in the feature region.

[0105] The example shown in Figure 10 above is just one example in this embodiment, and the method of controlling the reading method for the region of interest and the region of non-interest is not limited to this method.

[0106] Although the first to fourth embodiments have been described separately, combinations of these embodiments may also be implemented.

[0107] For example, the feature region detection unit 102 may have multiple units, such as feature region detection unit 102a and feature region detection unit 102b, and each feature region detection unit may detect different feature regions. As an example, feature region detection unit 102a detects the flicker region, and feature region detection unit 102b detects the subject region. Based on the detection result of the flicker region, the image sensor control unit 103 controls the exposure time as shown in Figure 3(c) to accumulate pixel information. Subsequently, based on the subject region detection result, the image sensor control unit 103 changes the readout decimation rate as shown in Figure 10. This makes it possible to reduce the amount of data while reducing the effect of flicker and without reducing the resolution in the subject region. In this way, multiple regions may be detected simultaneously, and shooting control may be performed using multiple detection results.

[0108] Furthermore, the image sensor unit 104 has a detection target switching means for switching the target to be detected by the feature region detection unit 102, and the target to be detected as a feature region may be switched. For example, the image sensor unit 104 has a storage unit for storing a trained model used for feature region detection, and the detection target may be switched by switching the trained model used by the feature region detection unit 102 according to the shooting settings and the target to be shot.

[0109] (Other embodiments) In the above embodiments, the imaging device according to the present invention was described as a personal digital camera, but it is not limited to this. That is, as long as it is equipped with an imaging function and an image synthesis function and has a user interface for setting the exposure time, the imaging device according to the present invention may be a portable device, a smartphone, or a network camera connected to a server. In addition, some of the processing described above may be performed by a portable device, a smartphone, or a network camera connected to a server.

[0110] The present invention can also be realized by supplying a program that implements one or more of the functions of this embodiment to a system or device via a network or recording medium, and by having one or more processors in the computer of that system or device read and operate the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0111] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0112] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist.

[0113] This embodiment includes the following configurations, methods, and programs. (Configuration 1) An imaging device comprising an image sensor unit having a plurality of pixels and a setting unit that sets first shooting conditions according to user instructions, wherein the imaging device has inside the image sensor unit: detection means for detecting a feature region based on pixel information obtained from the plurality of pixels, and control means for causing the pixels of the feature region to be photographed under a second shooting condition different from the first shooting condition, and the pixels of other regions other than the feature region to be photographed under the first shooting condition. (Configuration 2) The imaging apparatus according to Configuration 1, characterized in that at least one of the exposure time setting, gain setting, and pixel information readout method setting differs from the first shooting condition. (Configuration 3) The imaging apparatus according to Configuration 1 or 2, characterized in that the control means outputs pixel information from pixels of the feature region obtained when shooting under the second shooting conditions and pixel information from pixels of the other region obtained when shooting under the first shooting conditions as a single image file. (Configuration 4) The imaging apparatus according to any one of Configurations 1 to 3, characterized in that the control means detects a feature quantity in the feature region and determines the second shooting condition according to the feature quantity. (Configuration 5) The imaging apparatus according to Configuration 4, wherein the first shooting condition includes a first exposure time for pixels in the other region, the second shooting condition includes a second exposure time different from the first exposure time for pixels in the feature region, and the control means determines the second exposure time according to the first shooting condition and the feature quantity. (Configuration 6) The imaging apparatus according to Configuration 5, characterized in that the control means controls the exposure start timing and the readout timing so as to align the exposure centroids of the first and second exposure times. (Configuration 7) The imaging apparatus according to any one of Configurations 1 to 6, characterized in that each of the plurality of pixels has two or more photodiodes capable of independently reading out pixel information. (Configuration 8) The imaging apparatus according to Configuration 7, characterized in that the control means reads pixel information independently from each of the two or more photodiodes from the pixels of the feature region, and performs a focus position calculation from the phase difference of the independently read pixel information. (Configuration 9) The imaging apparatus according to any one of Configurations 2 to 6, characterized in that the control means reads out pixel information from the pixels of the feature region by downsampling at a predetermined downsampling rate. (Configuration 10) The imaging device according to any one of Configurations 1 to 9, wherein the detection means is also provided outside the image sensor, and the device further comprises a switching means for switching which detection means to use from among the detection means located inside and outside the image sensor. (Configuration 11) The imaging apparatus according to any one of Configurations 1 to 10, wherein the detection means further comprises a detection target switching means for switching the target to be detected as the feature region. (Method 1) A control method for an imaging device comprising an image sensor unit having a plurality of pixels and a setting unit that sets first shooting conditions according to user instructions, characterized in that the image sensor unit performs a detection step of detecting a feature region based on pixel information obtained from the plurality of pixels, and a control step of causing the pixels of the feature region to be photographed under a second shooting condition different from the first shooting condition, and the pixels of other regions other than the feature region to be photographed under the first shooting condition. (Program 1) A program for causing a computer to function as one of the means of an imaging device described in any one of configurations 1 to 11. [Explanation of symbols]

[0114] 100 Imaging device 101 pixel section 102 Feature Region Detection Unit 103 Image sensor control unit 104 Image sensor section 105 System Control Unit 106 Records Section 107 Display section 108 Operation section R301, R303, R305 Flicker Regions R302,R304,R306,R602,R604,R802 Normal area R601, R603 Low-luminance areas 801a~801d Photodiodes R803 Subject area

Claims

1. An imaging device comprising an image sensor unit having multiple pixels and a setting unit that sets first shooting conditions according to user instructions, A detection means for detecting a feature region based on pixel information obtained from the plurality of pixels, A control means that causes the pixels in the feature region to be photographed under a second shooting condition different from the first shooting condition, and the pixels in other regions other than the feature region to be photographed under the first shooting condition, An imaging device characterized by having the image sensor portion inside the image sensor portion.

2. The imaging apparatus according to claim 1, characterized in that the second shooting condition is different from the first shooting condition in at least one of the exposure time setting, gain setting, and pixel information readout method setting.

3. The imaging apparatus according to claim 1, characterized in that the control means outputs pixel information from pixels in the feature region obtained when shooting under the second shooting conditions and pixel information from pixels in the other region obtained when shooting under the first shooting conditions as a single image data.

4. The imaging apparatus according to claim 1, characterized in that the control means detects a feature quantity in the feature region and determines the second shooting condition according to the feature quantity.

5. The first shooting condition includes a first exposure time for the pixels in the other region. The second shooting condition includes a second exposure time different from the first exposure time for the pixels of the feature region, The imaging apparatus according to claim 4, wherein the control means determines the second exposure time according to the first shooting conditions and the feature quantity.

6. The imaging apparatus according to claim 5, characterized in that the control means controls the exposure start timing and the readout timing so as to align the exposure centroids of the first and second exposure times.

7. The imaging apparatus according to claim 1, characterized in that each of the plurality of pixels has two or more photodiodes capable of independently reading out pixel information.

8. The imaging apparatus according to claim 7, characterized in that the control means reads pixel information independently from each of the two or more photodiodes from the pixels of the feature region and performs a focus position calculation from the phase difference of the independently read pixel information.

9. The imaging apparatus according to claim 2, characterized in that the control means reads out pixel information from the pixels of the feature region by downsampling it at a predetermined downsampling rate.

10. The imaging apparatus according to claim 1, wherein the detection means is also provided outside the image sensor, and further comprises a switching means for switching which detection means to be used from among the detection means located inside and outside the image sensor.

11. The imaging apparatus according to claim 1, wherein the detection means further comprises a detection target switching means for switching the target to be detected as the feature region.

12. A control method for an imaging device comprising an image sensor unit having multiple pixels and a setting unit that sets first shooting conditions according to user instructions, In the aforementioned image sensor section, A detection step of detecting a feature region based on pixel information obtained from the plurality of pixels, A control step in which, among the plurality of pixels, the pixels in the feature region are photographed under a second shooting condition different from the first shooting condition, and the pixels in other regions other than the feature region are photographed under the first shooting condition, A control method characterized by performing the following.

13. A program for causing a computer to function as each of the means of the imaging apparatus described in claim 1.

Images