Imaging control device, imaging control method, and program

Abstract

To optimize exposure conditions in accordance with a distance to a subject.SOLUTION: An imaging control device includes acquisition means for acquiring distance information to a subject contained in a divided area obtained by dividing an imaging area, calculating means for calculating the distance to the subject contained in the divided area on the basis of the distance information to the subject contained in the divided area, and determining means for determining exposure conditions for the divided area containing the subject so that the exposure time is shortened when the distance calculated by the calculating means is shorter than that when it is longer.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging control device, an imaging control method, and a program.

Background Art

[0002] There is known a photographing apparatus that corrects subject blur in an image (Patent Document 1). In this photographing apparatus, when it is calculated that a subject is at a predetermined short distance and the shutter speed is slower than a predetermined shutter speed, a plurality of images are generated in response to a single imaging instruction. Then, based on a motion vector representing the movement of the subject in the image obtained by the image sensor, the plurality of images are superimposed so that subject blur in the image is corrected.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to optimize exposure conditions according to the distance to the subject.

Means for Solving the Problems

[0005] An imaging control device according to one aspect of the present invention includes: an acquisition unit that acquires distance information to a subject included in a divided region obtained by dividing an imaging region; a calculation unit that calculates the distance to the subject included in the divided region based on the distance information to the subject included in the divided region; and a determination unit that determines exposure conditions for the divided region including the subject so that the exposure time is shorter when the distance calculated by the calculation unit is short than when it is long. The calculation means calculates a distance distribution of a subject included in the image captured in the imaging region based on distance information to the subject included in the divided region and the position of the divided region. The determination means determines a range in which the exposure time of the divided region can be set based on the distance distribution calculated by the calculation means. The starting point for performing the process from the acquisition of the distance information to the determination of the exposure condition includes at least any one of the detection of a moving object in the image by image recognition, the detection of movement by a gyro sensor or an acceleration sensor, the instruction for enlarged display, and the designation of the cut-out range of the image. It is characterized by the above.

Effects of the Invention

[0006] According to one aspect of the present invention, the exposure conditions can be optimized according to the distance to the subject.

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention, and not all combinations of the features described in the embodiments are essential for the solution means of the present invention. The configuration of the embodiments can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environments, etc.). The technical scope of the present invention is determined by the scope of the claims and is not limited by the following individual embodiments.

[0009] <First Embodiment> FIG. 1 is a block diagram showing a configuration example of an imaging device according to the first embodiment. Among the functional modules of the imaging device 100 shown in FIG. 1, for the functions realized by software, a program for providing the functions of each functional module is stored in a memory such as a ROM (Read Only Memory). Then, the program is read into a RAM (Random Access Memory) and executed by a CPU (Central Processing Unit) to be realized. For the functions realized by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on an FPGA from a program for realizing the functions of each functional module. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as the FPGA and realized as hardware. Alternatively, it may be realized by an ASIC (Application Specific Integrated Circuit). Note that the configuration of the functional blocks shown in FIG. 1 is an example, and a plurality of functional blocks may constitute one functional block, or any one functional block may be divided into blocks that perform a plurality of functions.

[0010] In the present embodiment, even when a long-distance subject and a short-distance subject are mixed in the imaging angle of view, by determining the settable range of the exposure time for each pixel region of the imaging element based on the distance distribution from the subject, blurring amount corresponding to the subject distance is suppressed.

[0011] In FIG. 1, the imaging device 100 can set exposure conditions such as the shutter speed and analog gain for each divided region obtained by dividing the imaging region, and control the exposure conditions for each divided region to capture an image. The divided region may be composed of a single pixel or a plurality of pixels. When the divided region is composed of several pixels, the imaging region may be divided into blocks. The imaging device 100 may be used alone or mounted on a smartphone, a monitoring device, or the like.

[0012] In the following description, a single pixel or a plurality of pixels controlled under the same exposure conditions (exposure time and analog gain) may be referred to as a pixel region. The pixel region can be configured in a divided region obtained by dividing the imaging region. The number of pixels in each pixel region may be different. Further, in order to reduce the load of image processing, the imaging device 100 may collectively control a plurality of exposure regions having the same exposure conditions.

[0013] The imaging device 100 includes an imaging unit 101, an A / D (Analog / Digital) conversion unit 102, a signal processing unit 103, a D / A (Digital / Analog) conversion unit 104, an encoder unit 105, and a media I / F (interface) unit 106. The imaging device 100 also includes a CPU 107, a ROM 108, and a RAM 109. Furthermore, the imaging device 100 includes an imaging system control unit 110, an operation unit 111, a character generation unit 112, a display unit 113, a distance information acquisition unit 114, a distance distribution calculation unit 115, and an exposure condition determination unit 116.

[0014] The imaging unit 101 detects light from a subject for each pixel. The imaging unit 101 includes, for example, a zoom lens, a focus lens, an anti-shake lens, a diaphragm, a shutter, an optical low-pass filter, an IR (Infrared Rays) cut filter, a color filter, and an imaging sensor. The imaging sensor is, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor.

[0015] The A / D conversion unit 102 converts the detected amount of light from the subject into a digital value. The signal processing unit 103 performs signal processing on the digital value output from the A / D conversion unit 102 to generate a digital image. The signal processing performed by the signal processing unit 103 is, for example, demosaicking processing, white balance processing, and gamma processing. The D / A conversion unit 104 performs analog conversion on the digital image generated by the signal processing unit 103 for display.

[0016] The encoder unit 105 performs data compression processing on the digital image generated by the signal processing unit 103. For example, the encoder unit 105 performs processing such as data compression in the JPEG (Joint Photographic Experts Group) format. The media I / F unit 106 is an interface that connects the imaging device 100 to the media 117. The media 117 is, for example, a hard disk, a memory card, a CF (CompactFlash) card, an SD card, or a USB (Universal Serial Bus) memory, etc. The media 117 may be a PC (Personal Computer).

[0017] The CPU 107 performs the processing of the entire imaging device 100. The ROM 108 and the RAM 109 provide the CPU 107 with programs, data, work areas, etc. necessary for the processing. Also, when a control program necessary for the processing described later is stored in the ROM 108, it is once read into the RAM 109 and then executed by the CPU 107. Note that the CPU 107 may be a GPU (Graphics Processing Unit). The CPU 107 may be a single-core processor or a multi-core processor. The CPU 107 may have a function of operating as a neural network.

[0018] The operation unit 111 inputs instructions from the user. The operation unit 111 is, for example, buttons, a mode dial, a touch panel attached to the display unit 113, etc. The character generation unit 112 generates numbers, characters, symbols, graphics, etc. to be displayed on the display unit 113. The display unit 113 displays images such as captured images and GUI (Graphical User Interface). The display unit 113 is, for example, a CRT (Cathode-Ray Tube), a liquid crystal display, or a touch screen. Input via the touch screen can also be handled as input to the operation unit 111.

[0019] The distance information acquisition unit 114 acquires distance information to a subject included in a divided area obtained by dividing the imaging area. Further, the distance information acquisition unit 114 transmits the acquired distance information to the distance distribution calculation unit 115. At this time, the distance information acquisition unit 114 may include a distance information calculation unit that calculates the distance from the imaging unit 101 to the subject. The distance information can be calculated based on, for example, at least any one of image plane phase difference AF (Auto Focus) using an imaging optical system, contrast AF using an imaging optical system, distance measurement detection using light detection, identification of a subject by image recognition, and background difference.

[0020] For example, when calculating the distance using image plane phase difference AF, one or more pixels for image plane phase difference AF are arranged in each pixel area in the image sensor. Thereby, the distance information acquisition unit 114 can calculate the distance information to the subject for each pixel area. Further, in image plane phase difference AF, since the distance to the subject can be calculated from the phase difference appearing on the image plane, it is possible to calculate the distance information for each pixel area and for each frame.

[0021] In addition, the distance information acquisition unit 114 may obtain an evaluation value for each pixel region by contrast AF and estimate the distance to the subject from the in-focus position. Alternatively, a distance measurement technique using light detection typified by LIDAR (Light Detection and Ranging) may be used. Furthermore, the distance information acquisition unit 114 may identify the distance of the subject by image recognition. At this time, the distance information acquisition unit 114 estimates the distance for each pixel region from the size within the imaging angle of view of the subject. For example, the size of a specific subject such as the size of a person or a car is determined to a certain extent. When using the subject identification means, the distance information acquisition unit 114 estimates the distance to the subject based on the information of the subject whose size is determined to a certain extent. In addition, when there is a pixel region that cannot be detected by these distance information detection means, the distance information acquisition unit 114 predicts based on the distance information of the peripheral region of the corresponding pixel region, for example, by the average value of the peripheral region.

[0022] Alternatively, the user may arbitrarily set it, such as inputting distance information for each pixel region. In addition, the distance information acquisition unit 114 may perform foreground / background segmentation using image recognition and determine the relative distance between the foreground and the background. In this case, a special distance measurement mechanism such as a pixel for image plane phase difference AF is not required as the distance measurement means. Also, in foreground / background segmentation, a background difference technique may be used. This technique performs determination based on the temporal brightness change of the same pixel, rather than the brightness difference between pixels at the same time. Therefore, the distance information acquisition unit 114 can acquire distance information even in a state where the brightness gradation is impaired.

[0023] In addition, when the distance information acquisition unit 114 does not have a mechanism to acquire distance information for each pixel region, it may perform complementary processing such as linear interpolation from the change amount of distance information at multiple image positions. Alternatively, the distance information acquisition unit 114 may measure the distance by grouping a plurality of pixel regions. However, in this case, the exposure condition determination unit 116 needs to set the exposure condition for each region obtained by grouping a plurality of pixel regions. Further, when there are a plurality of pixels for measuring distances within a pixel region, the distance information acquisition unit 114 may calculate, from the obtained plurality of distance data, an average value, a median value, or a mode value of a histogram, etc., and determine it as the distance information of the corresponding pixel region.

[0024] The distance distribution calculation unit 115 calculates the distance to the subject included in the divided region based on the distance information to the subject included in the divided region obtained by dividing the imaging region. For example, the distance distribution calculation unit 115 can calculate the distance distribution based on the distance information acquired from the distance information acquisition unit 114 and the position within the imaging angle of view of the same exposure region corresponding to the distance information.

[0025] In addition, the distance distribution calculation unit 115 can calculate the distance distribution based on the closest or farthest distance information among the plurality of different distance information of the exposure regions controlled under the same exposure condition. Further, the distance distribution calculation unit 115 can calculate the distance distribution based on at least any one of the average value, the median value, and the mode value of the plurality of different distance information of the exposure regions controlled under the same exposure condition.

[0026] Furthermore, the distance distribution calculation unit 115 can set a threshold value of a reference distance and perform comparison to determine whether the distance information calculated by the distance information acquisition unit 114 is a short distance or a long distance. Here, the distance distribution calculation unit 115 can set the region with a short distance when the distance information is less than or equal to the threshold value, and set the region with a long distance when the distance information exceeds the threshold value. At this time, the distance distribution calculation unit 115 may set a plurality of threshold values.

[0027] The exposure condition determination unit 116 determines the exposure conditions for the divided regions based on the distance information to the subject included in the divided regions obtained by dividing the imaging region. Here, when the distance to the subject included in the divided region is short, the exposure condition determination unit 116 determines the exposure conditions for the divided region including the subject such that the exposure time is shorter than when the distance is long. At this time, the exposure condition determination unit 116 may determine the settable range of the exposure time for the divided region based on the distance distribution calculated by the distance distribution calculation unit 115.

[0028] Here, the exposure condition determination unit 116 can limit the upper limit of the exposure time for the divided region where the distance distribution calculated by the distance distribution calculation unit 115 is equal to or less than the threshold value. At this time, the exposure condition determination unit 116 may strengthen the limitation of the upper limit of the exposure time when the deviation between the distance distribution and the threshold value is small compared to when it is large.

[0029] In addition, the exposure condition determination unit 116 can limit the upper limit of the exposure time for the divided region where the distance distribution calculated by the distance distribution calculation unit 115 exceeds the threshold value. At this time, the exposure condition determination unit 116 may strengthen the limitation of the upper limit of the exposure time when the deviation between the distance distribution calculated by the distance distribution calculation unit 115 and the threshold value is large compared to when it is small.

[0030] Further, the exposure condition determination unit 116 may determine at least one of a threshold value for limiting the upper limit of the exposure time and the upper limit of the exposure time based on the distance information calculated by the distance distribution calculation unit 115. Further, the exposure condition determination unit 116 may determine at least one of a threshold value for limiting the upper limit of the exposure time and the upper limit of the exposure time based on the size of the subject detected based on image recognition and the range of the imaging angle of view. Further, the exposure condition determination unit 116 may determine at least one of a threshold value for limiting the upper limit of the exposure time and the upper limit of the exposure time based on the magnitude of the movement vector of the subject in the image. Further, the exposure condition determination unit 116 may switch the region for limiting the upper limit of the exposure time according to the display format of the enlarged display. Further, the exposure condition determination unit 116 may determine the exposure amount so as to achieve proper exposure by analog gain with respect to the setting of the exposure time. Further, the exposure condition determination unit 116 may cause all the exposure regions respectively controlled under the same exposure conditions to be imaged in a single frame.

[0031] The imaging system control unit 110 controls the imaging system instructed by the CPU 107. The imaging system control unit 110 sets exposure conditions such as the shutter speed and analog gain for each pixel or region for the imaging unit 101. At this time, the imaging system control unit 110 can apply the exposure conditions determined by the exposure condition determination unit 116 to the imaging unit 101. Further, the imaging system control unit 110 also performs controls such as focus adjustment, shutter opening / closing, and aperture adjustment. Note that the imaging device 100 can include various components other than the above-described components, but the description thereof is omitted.

[0032] Here, even when the moving speed of the subject (moving amount / exposure time) is the same, as seen in the image, the moving amount per pixel increases as the distance to the subject decreases. Therefore, the blur amount of the moving object at close range is larger than that of the moving object at long range. At this time, in order to suppress the blur amount of the moving object at close range, if the exposure time is uniformly limited to a short time for the entire image, the region of the moving object at long range is also imaged with high gain, and the noise of the entire image increases.

[0033] At this time, the exposure condition determination unit 116 can obtain the distance distribution for each region and limit the upper limit of the exposure time for each region to the short-second side based on the distance distribution. Therefore, even when a subject at a long distance and a subject at a short distance are mixed within the imaging angle of view, the upper limit of the exposure time for the region where the subject at a long distance is located can be limited to the short-second side without limiting the upper limit of the exposure time for the region where the subject at a short distance is located to the short-second side. As a result, it is possible to suppress an increase in noise in the region where the subject at a long distance is located and suppress the amount of blur in the region where the subject at a short distance is located.

[0034] According to the first embodiment described above, the imaging device 100 determines the exposure conditions for the divided regions such that the exposure time is shorter when the distance to the subject included in the divided region obtained by dividing the imaging region is short than when it is long. Thereby, the imaging device 100 can prevent an increase in noise in the entire image and suppress blur of the subject.

[0035] In the example of FIG. 1, the distance information acquisition unit 114, the distance distribution calculation unit 115, and the exposure condition determination unit 116 are shown as blocks separate from the CPU 107. The imaging control process executed by the distance information acquisition unit 114, the distance distribution calculation unit 115, and the exposure condition determination unit 116 may be executed by the CPU 107 as a part of the functions realized by the CPU 107.

[0036] FIG. 2 is a diagram showing an example of an imaging image according to the first embodiment. In FIG. 2, an example is shown in which there are four exposure regions that are each exposed under the same exposure conditions within the imaging angle of view. In the following description, an exposure region that is exposed under the same exposure conditions may be referred to as the same exposure region.

[0037] In FIG. 2, the same exposure regions A1, A2, B1, and B2 are set at four locations in the imaging image 200. In this imaging image 200, the same exposure region located in the upper left is A1, the same exposure region located in the upper right is A2, the same exposure region located in the lower left is B1, and the same exposure region located in the lower right is B2.

[0038] At this time, assume that scenes with a long distance to the subject are imaged in the same exposure areas A1 and A2, and scenes with a short distance to the subject are imaged in the same exposure areas B1 and B2. Also, assume that the subject 201 is located at a short distance and the subject 202 is located at a long distance. For this reason, even for the same vehicle type, the subject 201 is imaged larger than the subject 202. Also, assume that the moving speeds of the subjects 201 and 202 are equal.

[0039] In addition, in FIG. 2, the case where the same exposure areas A1, A2, B1, and B2 are present at four locations in the captured image 200 is shown, but it is sufficient that there are two or more same exposure areas. However, the number of pixels of the image sensor becomes the upper limit number of the same exposure amount. Also, in FIG. 2, the vehicle subjects 201 and 202 are cited as the objects of the distance to the subject, but the subject is not limited to the vehicle. For example, it may be a background such as a road, the sky, a landscape, or a building, or it may be a person or an animal other than the illustrated vehicle.

[0040] FIG. 3 is a diagram showing the relationship between the imaging angle of view according to the first embodiment and the size and distance of the subject. In FIG. 3, in order to calculate the distance distribution of the distance to the subject, the distance distribution calculation unit 115 sets a determination criterion for whether the distance to the subject is short or long. At this time, the distance distribution calculation unit 115 calculates a determination reference distance D [m] that is a threshold for determining whether the distance to the subject is short or long, assuming that the subject is at the distance of the threshold that is the determination criterion.

[0041] Let the horizontal angle of the imaging angle of view be the imaging angle Ra [rad], half of the angle be the half imaging angle Rb [rad], the horizontal length of the subject be the subject length X [m], and the length of the horizontal imaging range at the determination reference distance D be the imaging range H [m]. At this time, the distance distribution calculation unit 115 determines whether the subject is close or far depending on what percentage of the size of the subject length X is within the imaging range H. If the ratio of the subject length X in the imaging range H is the determination reference value Y [%], the determination reference value Y can be given by the following equation (1). Also, the determination reference distance D [m] that is a threshold for determining whether the distance to the subject is short or long can be given by the following equation (2).

[0042] Y = X / H × 100 H = (X × 100) / Y ···(1) tan(Rb) = ((H / 2) / D) tan(Rb) = ((X × 100) / 2) / (Y × D) tan(Rb) = (X × 50) / (Y × D) ···(2)

[0043] For example, when the user sets a threshold value, assume that a car is taken as the subject. At this time, if the distance distribution calculation unit 115 sets the total length of the car in the horizontal direction as X = 5 m and can image the car at a size where the angle of view Y is 25% or more, it determines that the distance to the subject is close. If it is smaller than that, it determines that the distance to the subject is far. Here, assume that the imaging angle of view Ra is π / 2 [rad] and the half imaging angle Rb is π / 4 [rad]. At this time, from equations (1) and (2), the determination reference distance D serving as the threshold value can be given by the following equation (3). In the case of the above example, the determination reference distance D is 10 m.

[0044] tan(π / 4 [rad]) = (5 [m] × 50) / (25 [%] × D [m]) 1 = 10 [m] / D [m] D = 10 [m] ···(3)

[0045] The distance distribution calculation unit 115 can determine whether the subject is closer or farther than the determination reference distance D for each same exposure region by comparing the distance information calculated by the distance information acquisition unit 114 with respect to this determination reference distance D. The distance distribution calculation unit 115 creates a distance distribution based on this determination result. In FIG. 3, although the horizontal direction is taken as an example as an overhead view of the imaging angle of view, the vertical direction or the diagonal direction may also be used.

[0046] FIG. 4 is a diagram showing an example of setting the distance distribution according to the first embodiment. In FIG. 4, the places recognized as having a close distance are shown in white, and the places recognized as having a far distance are shown in black. In FIG. 4, the distance distribution calculation unit 115 generates a distance distribution by comparing the distance information acquired from the distance information acquisition unit 114 with the threshold value shown in FIG. 3 for the imaging region 200 in FIG. 2. At this time, since the same exposure regions A1 and A2 exceed the threshold value, they are determined as the long-distance region 4A, and since the same exposure regions B1 and B2 are below the threshold value, they are determined as the short-distance region 4B.

[0047] Note that in FIG. 4, it is assumed that distances can be measured at a plurality of pixel positions within the same exposure regions A1, A2, B1, and B2. At this time, when both a short-distance subject and a long-distance subject are imaged within the same exposure region, the distance distribution calculation unit 115 preferably determines that the short-distance subject with strict restrictions is being imaged and creates a distance distribution. However, the distance distribution calculation unit 115 may also obtain a distance histogram within the same exposure region from the results of measuring distances at a plurality of pixel positions, and create a distance distribution assuming that the subject is located at the distance with the highest ratio. Additionally, the distance distribution calculation unit 115 may determine the distance to the subject within the same exposure region by using, for example, the median or average value of the distance distribution. Further, the distance distribution calculation unit 115 may calculate a threshold value as a relative value from the distance information in the image. For example, the distance distribution calculation unit 115 may use the median value of the distance information within the entire angle of view as the threshold value, and define a region with a distance closer than the threshold value as the short-distance region and a region with a distance farther than the threshold value as the long-distance region.

[0048] FIG. 5 is a flowchart showing the imaging process according to the first embodiment. Note that each step in FIG. 5 is realized by the distance distribution calculation unit 115 and the exposure condition determination unit 116 reading and executing the program stored in the storage unit of the imaging device 100 in FIG. 1. Further, at least a part of the flowchart shown in FIG. 5 may be realized by hardware. When realized by hardware, for example, a dedicated circuit may be automatically generated on the FPGA from the program for realizing each step by using a predetermined compiler. Also, a Gate Array circuit may be formed in the same manner as the FPGA and realized as hardware. Further, it may be realized by an ASIC. In this case, each block in the flowchart shown in FIG. 5 can be regarded as a hardware block. Note that a plurality of blocks may be combined into one hardware block, or one block may be configured as a plurality of hardware blocks.

[0049] In addition, in the imaging process of FIG. 5, it is assumed that the subject moves. When the moving speed of the subject is the same, for the angle of view, the blur amount of the nearby subject is large and the blur amount of the distant subject is small. Therefore, the imaging device 100 can limit the upper limit of the exposure time for the nearby subject and perform imaging in a short time, so that even for a nearby subject, the blur amount can be reduced for imaging.

[0050] In S501 of FIG. 5, the distance distribution calculation unit 115 acquires distance information from the distance acquisition unit 114 for each same exposure region. In the example of FIG. 2, the same exposure regions are the regions of A1, A2, B1, and B2.

[0051] Next, in S502 of FIG. 5, the distance distribution calculation unit 115 creates a distance distribution of the distance to the subject based on the distance information acquired from the distance acquisition unit 114. For example, the distance distribution calculation unit 115 can create the distance distribution of FIG. 4 for the same exposure regions A1, A2, B1, and B2 in FIG. 2. Thereby, the distance distribution calculation unit 115 can associate the determination result of whether the distance to the subject in the same exposure region is closer or farther than the threshold value for each same exposure region.

[0052] Next, in S503 of FIG. 5, the exposure condition determination unit 116 initializes the region position in order to set the exposure conditions for each region. The exposure condition determination unit 116 temporarily sets the initial position as the same exposure region A1. When moving to the next region, it is desirable for the exposure condition determination unit 116 to move horizontally in alignment with the scanning direction of the image sensor and then move vertically. For example, in the example of FIG. 2, the exposure condition determination unit 116 moves through the same exposure regions in the order of A1, A2, B1, and B2.

[0053] Next, in S504 of FIG. 5, the exposure condition determination unit 116 refers to the distance distribution in FIG. 4 and determines whether the distance to the subject in the corresponding identical exposure area is equal to or less than a threshold value. If the distance to the subject in the identical exposure area is equal to or less than the threshold value, the exposure condition determination unit 116 proceeds to S505; if it is greater than the threshold value, the exposure condition determination unit 116 proceeds to S506.

[0054] In S505 of FIG. 5, since the distance to the subject is short, in order to suppress the amount of blur of the subject, the exposure condition determination unit 116 restricts the upper limit of the exposure time so that the exposure time of the exposure area including the subject becomes short, and sets the exposure conditions.

[0055] Next, in S506 of FIG. 5, since the distance to the subject is long, the upper limit of the exposure time of the exposure area including the subject is not set depending on the distance, and the optimal exposure conditions are set. As a result, the exposure condition determination unit 116 can make the value of the analog gain of the exposure area including the subject with a long distance smaller than the value set in S505, and can suppress an increase in noise. After determining the settable range of the exposure time in S505 or S506, the exposure condition determination unit 116 proceeds to S507.

[0056] In S507 of FIG. 5, the exposure condition determination unit 116 sets the exposure conditions for each identical exposure area. At this time, the exposure condition determination unit 116 determines the range of the set value so as to satisfy the exposure conditions (exposure time) set in S505 or S506. Here, it is desirable that the exposure condition determination unit 116 sets not only the exposure time but also the analog gain (ISO sensitivity) so as to achieve proper exposure after referring to the luminance value of the subject. Note that when suppressing the amount of blur of the subject, the exposure condition determination unit 116 does not have to restrict the lower limit at which the exposure time becomes short. After setting the exposure conditions, the exposure condition determination unit 116 proceeds to S508.

[0057] Next, in S508 of FIG. 5, the exposure condition determination unit 116 refers to the position of the current identical exposure area and determines whether the exposure conditions for all the identical exposure areas have been set. If the exposure condition determination unit 116 has set the exposure conditions for all the identical exposure areas, it proceeds to S509; if not, it proceeds to S510.

[0058] In S509 of FIG. 5, the exposure condition determination unit 116 moves to the identical exposure area at the next position and returns to S504. In S510 of FIG. 5, the imaging system control unit 110 causes the imaging unit 101 to perform imaging based on the exposure conditions set in S507.

[0059] As shown in the imaging process of FIG. 5, by setting an upper limit for the exposure time according to the distance distribution, it is possible to capture an image with a reduced amount of blur in the area where the upper limit of the exposure time is restricted. Thereby, even when a subject at a long distance and a subject at a short distance are mixed within the imaging angle of view of the imaging device 100, the imaging device 100 can acquire an image with motion blur suppressed according to the distance. Furthermore, for a subject at a long distance, since the upper limit of the exposure time is not restricted, the imaging device 100 can set a relatively long exposure time, suppress the analog gain, and perform imaging with a high S / N (signal-to-noise ratio) and an appropriate exposure amount.

[0060] Note that in the above-described first embodiment, the case where the subject is a moving object and there is motion blur of the subject is taken as an example, but it is not limited to this case. For example, blur of the image may also occur due to camera shake of the imaging device 100 or vibration at the installation location. The first embodiment is also applicable to such blur of the imaging device 100 itself. However, the phenomenon in which the amount of blur varies according to the distance shown in the first embodiment is caused by blur in the shift direction of the imaging device 100. At this time, since the amount of blur of the subject at a short distance becomes large, the imaging device 100 suppresses blur for the subject at a short distance.

[0061] In addition, the imaging device 100 can capture all the same exposure areas at the same timing (one frame), and can prevent the deviation of the imaging timing between the areas. That is, since imaging can be performed at the same timing uniformly across the entire image, even if the exposure times are different, it is possible to prevent the subject from being missed and the occurrence of double images as compared with the case of capturing a plurality of images for each same exposure area.

[0062] Also, although the imaging device 100 may always perform the imaging process of FIG. 5 while it is operating, it is not essential to always perform the imaging process of FIG. 5 while the imaging device 100 is operating. At this time, the starting point for performing the imaging process of FIG. 5 may include at least any one of the detection of a moving object in the image by image recognition, the detection of movement by a gyro sensor or an acceleration sensor, the instruction for enlarged display, and the designation of the image cutout range. For example, the imaging process of FIG. 5 may be performed only when a moving object or a specific subject is captured in the image using image recognition processing such as optical flow, edge detection, or background difference. Also, the imaging process of FIG. 5 may be performed when blur of the imaging device 100 is detected from the values of the gyro sensor or the acceleration sensor attached to the imaging device 100. Also, the imaging device 100 may perform the imaging process of FIG. 5 not every frame but every several frames.

[0063] Also, in the imaging process of FIG. 5, although the case of one threshold value has been described, there may be a plurality of threshold values. In that case, the exposure condition determination unit 116 determines which distance among the plurality of threshold values the distance to the subject in the same exposure area is located at, and sets the upper limit of the exposure time. At this time, it is desirable that the exposure condition determination unit 116 sets the exposure conditions such that the upper limit becomes shorter as the determination reference distance D exceeds a larger threshold value.

[0064] Also, in the imaging process of FIG. 5, the upper limit of the exposure time is restricted so that the exposure time becomes shorter. However, the exposure condition determination unit 116 may refer to the distance information to the subject in order to determine how much to restrict the upper limit. In that case, it is desirable that the exposure condition determination unit 116 strengthens the restriction of the upper limit of the exposure time as the distance to the subject is farther based on the reference of the distance information. Also, the exposure condition determination unit 116 may set the upper limit of the exposure time with reference to not only the distance information but also the size and moving speed of the subject. Also, the method for determining the distance threshold shown in S505 of FIGS. 3 and 5 is an example and is not limited to this method. For example, the designer or user may arbitrarily set the distance threshold.

[0065] Also, in the imaging process of FIG. 5, it was assumed that the upper limit of the exposure time is not provided when the distance to the subject is close. However, if the upper limit is less restrictive than the value set in S505, the upper limit of the exposure time may be set. Furthermore, the imaging device 100 may separately have an upper limit of the exposure time determined by the performance of the sensor, the performance of the image processing, the frame rate, and the like.

[0066] Also, when an area where the distance to the subject is close and an area where the distance is far are adjacent, the exposure time may change extremely due to the setting of the upper limit of the exposure time, resulting in an image with a large sense of discomfort when viewed as a single image. For this reason, the exposure condition determination unit 116 may limit the step of the exposure time between adjacent identical exposure areas.

[0067] (Second Embodiment) In the first embodiment described above, an example where the amount of blur in an area with a short distance is large was given, and a method for restricting the upper limit of the exposure time for an area where the distance to the subject is short was shown. In the second embodiment, a method for restricting the upper limit of the exposure time for an area where the distance to the subject is long is shown. In the second embodiment, cases where the amount of blur of a subject with a long distance is visually prominent or has a large influence from the viewpoint of image recognition are taken as examples. In the second embodiment, it is assumed that the image is enlarged by image processing such as digital zoom or image cropping. Also, in the second embodiment, a method for suppressing the blur of the image in the rotational blur of the imaging device 100 is also shown.

[0068] FIG. 6 is a diagram showing an example of a captured image according to the second embodiment. Note that FIG. 6(a) shows an image before enlargement of the captured image, and FIG. 6(b) shows an image after enlargement of the captured image. In FIG. 6(a), the same exposure regions A1, A2, B1, and B2 are set at four locations in the captured image 600.

[0069] Here, when the cutout range 602 is specified on the captured image 600 in FIG. 6(a), the imaging device 100 can cut out the image of the cutout range 602 from the captured image 600 and generate an enlarged image 601 in FIG. 6(b) obtained by enlarging the image of the cutout range 602. At this time, the captured image 600 and the enlarged image 601 are displayed by the display unit 113 and can be visually recognized.

[0070] Here, since the cutout range 602 shows a distant subject, the amount of blur per pixel of the image sensor is smaller than that of a nearby subject. However, the amount of blur with respect to the size of the enlarged image 601 cut out from the captured image 600 and enlarged for display is larger than the amount of blur with respect to the size of the captured image 600. That is, by changing the display size by cutting out, the apparent amount of blur changes. At this time, the exposure condition determination unit 116 can suppress the amount of blur also in the enlarged image 601 by limiting the upper limit of the exposure time for a region where the distance to the subject is far. Note that it is desirable to determine the reference distance serving as a threshold value from the display size of the enlarged image 601 instead of the imaging angle of view.

[0071] According to the second embodiment described above, even when the distance to the subject included in the divided region obtained by dividing the imaging region is long, the imaging device 100 determines the exposure conditions of the divided region so that the exposure time is shorter when the magnification of the subject is large than when it is small. Thereby, even when the cut-out image is enlarged, the imaging device 100 can suppress blurring of the subject while preventing an increase in noise of the entire image.

[0072] In the imaging process of the first embodiment described above, in S504 of FIG. 5, the exposure condition determination unit 116 sets the area close to the threshold value as the area where the upper limit of the exposure time is restricted, and the area far from the threshold value as the area where the upper limit of the exposure time is not restricted. In the second embodiment, the exposure condition determination unit 116 can set the area close to the threshold value as the area where the upper limit of the exposure time is not restricted, and the area far from the threshold value as the area where the upper limit of the exposure time is restricted. That is, in the second embodiment, when the distance to the subject exceeds the threshold value, the exposure condition determination unit 116 proceeds to S505, and when the distance to the subject is equal to or less than the threshold value, it proceeds to S506. Other processes in the second embodiment are the same as those in the first embodiment, and the description thereof is omitted.

[0073] The difference between the first embodiment and the second embodiment is whether the distance between the subject in the area where the upper limit of the exposure time is restricted and the subject is closer or farther than the threshold value. Regarding other points, the second embodiment can be processed in the same manner as the first embodiment.

[0074] In this way, even when there are a mixture of subjects at a long distance and subjects at a short distance within the imaging angle of view, and the amount of blur of the subject at a long distance is prominent, the imaging device 100 can acquire an image with motion blur suppressed according to the distance.

[0075] In addition, in FIG. 6, when the imaging device 100 displays only the enlarged image 601 on the display unit 113, it is desirable for the exposure condition determination unit 116 to limit the upper limit of the exposure time for regions at a long distance. Thereby, the imaging device 100 can display an image with a small amount of blur in the enlarged image 601. On the other hand, when the imaging device 100 simultaneously displays the captured image 600 and the enlarged image 601 on the display unit 113, it is desirable to allow the user to select which image to prioritize. When prioritizing the enlarged image 600, the imaging device 100 can obtain an image with a small amount of blur in the captured image 600 by limiting the upper limit of the exposure time for regions close to the subject. When prioritizing the enlarged image 601, the imaging device 100 can display an image with a small amount of blur in the enlarged image 601 by limiting the upper limit of the exposure time for regions at a long distance.

[0076] Also, the imaging device 100 can reduce the load of image processing during the period when the enlarged display is not performed by performing the imaging process of the second embodiment at the timing when the user displays the enlarged image or specifies the cutout range. Further, when the cutout range is specified, the imaging device 100 can reduce the load of image processing outside the cutout range by applying the imaging process of the second embodiment only to the same exposure area corresponding to the cutout range.

[0077] Also, in the second embodiment, the imaging process in the enlarged display has been described, but it is not limited to the enlarged display. For example, in the image recognition process, the imaging process of the second embodiment can also be applied. Since distant subjects occupy a smaller proportion of the imaging angle of view and recognition is performed with a small number of pixels, it is disadvantageous from the perspective of image recognition. At this time, if the subject in the captured image is blurred, the recognition accuracy further decreases. Therefore, in the image recognition process, it is desirable to suppress the amount of blur for distant subjects that are disadvantageous for image recognition.

[0078] Also, even when the imaging device 100 itself shakes in the rotational direction, since a subject farther away is imaged with a smaller number of pixels, the amount of blur with respect to the size of the subject increases as the distance increases. Therefore, even when the imaging device 100 itself is shaking in this way, it is desirable to apply the imaging process of the second embodiment to suppress the amount of blur of a distant subject.

[0079] (Third Embodiment) In the above-described first and second embodiments, a method for determining the determination reference distance D, which is the threshold value of the distance information, has been shown. Within the imaging angle of view, the moving speeds of the subjects are not always the same. Therefore, it is desirable to set the threshold value of the distance information and the upper limit of the exposure time according to the moving speed of the subject. In the third embodiment, a method for determining the threshold value of the distance information based not only on the distance information but also on the moving speed of the subject will be shown.

[0080] FIG. 7 is a diagram showing an example of an imaging image according to the third embodiment. Note that FIG. 7 shows an example of an image in which subjects with different moving speeds are imaged.

[0081] The imaging image 700 in FIG. 7 shows the same scene as the imaging image 200 in FIG. 2. Here, the subjects 201 and 202 in FIG. 2 correspond to the subjects 701 and 702 in FIG. 7. However, in the imaging image 200 in FIG. 2, the moving speeds of the subjects 201 and 202 are assumed to be equal, while in the imaging image 700 in FIG. 7, the moving speed of the subject 702 is greater than the moving speed of the subject 701. In FIG. 7, the magnitudes of the moving vectors of the subjects 701 and 702 are indicated by the sizes of the arrows.

[0082] Also, the same exposure regions A1 and A2 are regions far from the subject, and the subject 702 moving at high speed is imaged in the same exposure region A1. The same exposure regions B1 and B2 are regions close to the subject, and the subject 701 moving at low speed is imaged in the same exposure region B2. Also, as the distance distribution, as in FIG. 4, it is assumed that the same exposure regions A1 and A2 are located in the same distance distribution, and the same exposure regions B1 and B2 are located in the same distance distribution.

[0083] The imaging device 100 can calculate the moving speeds of the subjects 701 and 702 using optical flow by continuously imaging at a predetermined frame rate. At this time, regarding the imaging area of the subject 701, when the subject 701 moves slowly, it is not necessary to limit the upper limit of the exposure time. On the other hand, regarding the imaging area of the subject 702, when the subject 702 moves fast, it is desirable to limit the upper limit of the exposure time. However, the imaging device 100 can obtain an image with motion blur suppressed without delay even when the subject 701 moves to another area by setting the upper limit of the exposure time for an area with a short distance. That is, the same as in the first and second embodiments, a threshold is provided for the distance information.

[0084] FIG. 8 is a flowchart showing the imaging process according to the third embodiment. Since the imaging process in FIG. 8 is the same as the imaging process in FIG. 5 except for S804, S811, and S812, the differences from the imaging process in FIG. 5 will be described.

[0085] In S801 to S803 and S805 to S810 in FIG. 8, the same processing as in S501 to S503 and S505 to S510 in FIG. 5 is performed. After the processing in S802 is completed, the exposure condition determination unit 116 proceeds to S811.

[0086] In S811, the exposure condition determination unit 116 calculates the movement vector of the subject within the imaging angle of view using optical flow from the captured image. At this time, the imaging device 100 continuously images at a predetermined frame rate. Therefore, the time between frames is known to the imaging device 100. Also, the imaging device 100 can compare the images between frames to calculate the number of pixels the subject has moved, and further calculate the amount of movement per frame from the distance information of the distance information acquisition means 114. That is, since the imaging device 100 knows the movement time and movement distance of the subject between frames, it can calculate the movement speed of the subject.

[0087] Next, in S812, the exposure condition determination unit 116 determines a distance threshold for setting the upper limit of the exposure time from the movement vector of the subject calculated in S811 and the distance distribution. For example, even when the subject is located at the same distance, the exposure condition determination unit 116 can set the threshold so that the upper limit of the exposure time of the area including the subject becomes smaller when the magnitude of the movement vector of the subject is large. At this time, similar to the first embodiment, in the area where the distance from the subject is equal to or less than the threshold, it can be assumed that the moving speed of the captured subject is fast and the amount of blur is large.

[0088] Next, in S804, if the distance of the corresponding same exposure area is equal to or less than the threshold determined in S812, the exposure condition determination unit 116 proceeds to S805 and restricts the upper limit of the exposure time. On the other hand, if the distance of the corresponding same exposure area is greater than the threshold determined in S812, the exposure condition determination unit 116 proceeds to S806 and does not restrict the upper limit of the exposure time. Thereafter, the procedure is the same as that in FIG. 5.

[0089] Thereby, even when a subject at a long distance and a subject at a short distance are mixed within the imaging angle of view of the imaging device 100 and the moving speeds of the subjects are different, the imaging device 100 can acquire an image with motion blur suppressed according to the distance. Also, in the above description, it is assumed that the amount of blur is large at distances equal to or less than the threshold, but it is also applicable in the case where the amount of blur is large at distances exceeding the threshold, similar to the second embodiment. When a plurality of moving objects are located within the same exposure area, it is desirable that the exposure condition determination unit 116 refers to the moving speed of the subject with the fastest moving speed.

[0090] According to the third embodiment described above, even when the distance to the subject included in the divided area obtained by dividing the imaging area is long, the imaging device 100 determines the exposure conditions of the divided area so that the exposure time becomes shorter when the moving speed of the subject is fast than when it is slow. Thereby, even when subjects with different moving speeds are mixed, the imaging device 100 can suppress blur of the subject while preventing an increase in noise of the entire image.

[0091] <Other Embodiments> The present invention may supply a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium. Then, one or more functions of the above-described embodiments can also be realized by a process in which one or more processors in a computer of the system or apparatus read and execute the program. Further, it can also be realized by a circuit (for example, FPGA or ASIC) that realizes one or more functions.

Description of Reference Numerals

[0092] 101 Imaging unit, 102 A / D conversion unit, 103 Signal processing unit, 104 D / A conversion unit, 105 Encoder unit, 106 Media I / F unit, 107 CPU, 108 ROM, 109 RAM, 110 Imaging system control unit, 111 Operation unit, 112 Character generation unit, 113 Display unit, 114 Distance information acquisition unit, 115 Distance distribution calculation unit, 116 Exposure condition determination unit

Claims

1. acquisition means for acquiring distance information to a subject included in a divided area obtained by dividing an imaging area; calculation means for calculating the distance to a subject included in the divided area based on the distance information to the subject included in the divided area; determination means for determining the exposure condition of the divided area including the subject such that the exposure time becomes shorter when the distance calculated by the calculation means is short than when it is long; The calculation means calculates a distance distribution of a subject included in an image captured in the imaging area based on the distance information to the subject included in the divided area and the position of the divided area; The determination means determines a range in which the exposure time of the divided area can be set based on the distance distribution calculated by the calculation means; The starting point for performing the process from the acquisition of the distance information to the determination of the exposure condition includes at least one of detection of a moving object in the image by image recognition, detection of movement by a gyro sensor or an acceleration sensor, an instruction for enlarged display, and designation of a cutout range of the image. An imaging control device characterized by this.

2. The imaging control device according to claim 1, wherein the determination means limits an upper limit of the exposure time of a divided area in which the distance distribution calculated by the calculation means is equal to or less than a threshold value.

3. The imaging control device according to claim 2, wherein the determination means strengthens the limitation of the upper limit of the exposure time when the deviation between the distance distribution and the threshold value is small compared to when it is large.

4. The imaging control device according to claim 1, wherein the determination means limits an upper limit of the exposure time of a divided area in which the distance distribution calculated by the calculation means exceeds a threshold value.

5. The imaging control device according to claim 4, wherein the determination means strengthens the limitation of the upper limit of the exposure time when the deviation between the distance distribution and the threshold value is large compared to when it is small.

6. The imaging control device according to any one of claims 2 to 5, wherein the determining means sets a plurality of the threshold values.

7. The distance information is calculated based on at least one of image plane phase difference AF (Auto Focus) using an imaging optical system, contrast AF using an imaging optical system, distance detection using light detection, identification of a subject by image recognition, and background difference. The imaging control device according to any one of claims 1 to 6.

8. The imaging control device according to any one of claims 1 to 7, wherein the determining means determines at least one of a threshold value for limiting an upper limit of the exposure time and the upper limit of the exposure time based on the distance information.

9. An acquisition means for acquiring distance information to a subject included in a divided area obtained by dividing an imaging area; An arithmetic means for calculating a distance to a subject included in the divided area based on the distance information to the subject included in the divided area; Determining means for determining an exposure condition of the divided area including the subject such that an exposure time is shorter when the distance calculated by the arithmetic means is short than when it is long; The arithmetic means calculates a distance distribution of a subject included in an image captured in the imaging area based on the distance information to the subject included in the divided area and the position of the divided area. The determining means determines a range in which the exposure time of the divided area can be set based on the distance distribution calculated by the arithmetic means. An imaging control device, characterized in that at least one of a threshold value for limiting an upper limit of the exposure time and the upper limit of the exposure time is determined based on the size of a subject detected based on image recognition and the range of an imaging angle of view.

10. An acquisition means for acquiring distance information to a subject included in a divided area obtained by dividing an imaging area; An arithmetic means for calculating a distance to a subject included in the divided area based on the distance information to the subject included in the divided area; When the distance calculated by the calculation means is short, determination means for determining the exposure conditions of the divided region including the subject so that the exposure time is shorter than when the distance is long. The calculation means calculates a distance distribution for a subject included in the image captured in the imaging region based on the distance information to the subject included in the divided region and the position of the divided region. The determination means determines a range in which the exposure time of the divided region can be set based on the distance distribution calculated by the calculation means. An imaging control device, characterized in that at least one of a threshold value for limiting an upper limit of the exposure time and the upper limit of the exposure time is determined based on a magnitude of a movement vector of a subject in the image.

11. The imaging control device according to any one of claims 1 to 10, wherein the calculation means calculates a distance distribution based on the distance information of the closest distance or the farthest distance among a plurality of different distance information of exposure regions controlled under the same exposure conditions.

12. The imaging control device according to any one of claims 1 to 11, wherein the calculation means calculates a distance distribution based on at least one of an average value, a median value, and a mode value of a plurality of different distance information of exposure regions controlled under the same exposure conditions.

13. The imaging control device according to any one of claims 1 to 12, wherein the determination means causes all exposure regions respectively controlled under the same exposure conditions to be imaged in a single frame.

14. Acquisition means for acquiring distance information to a subject included in a divided region obtained by dividing an imaging region, Calculation means for calculating a distance to a subject included in the divided region based on the distance information to the subject included in the divided region, When the distance calculated by the calculation means is short, determination means for determining the exposure conditions of the divided region including the subject so that the exposure time is shorter than when the distance is long. The calculation means calculates a distance distribution of a subject included in the image captured in the imaging region based on the distance information to the subject included in the divided region and the position of the divided region. The determination means determines a settable range of the exposure time of the divided region based on the distance distribution calculated by the calculation means. An imaging control device characterized by switching a region that limits the upper limit of the exposure time according to the display format of the enlarged display.

15. The determination means determines an exposure amount so as to achieve appropriate exposure by an analog gain with respect to the setting of the exposure time, according to any one of claims 1 to 14. The imaging control device described in the item.

16. An acquisition step of acquiring distance information to a subject included in a divided region obtained by dividing an imaging region; A calculation step of calculating a distance to a subject included in the divided region based on the distance information to the subject included in the divided region; A determination step of determining an exposure condition of the divided region so that the exposure time is shorter when the calculated distance is short than when it is long, comprising: In the calculation step, a distance distribution of a subject included in the image captured in the imaging region is calculated based on the distance information to the subject included in the divided region and the position of the divided region. In the determination step, a settable range of the exposure time of the divided region is determined based on the distance distribution calculated in the calculation step. The starting point for performing the process from the acquisition of the distance information to the determination of the exposure condition includes at least any one of the detection of a moving object in the image by image recognition, the detection of movement by a gyro sensor or an acceleration sensor, the instruction of enlarged display, and the designation of the cut-out range of the image. An imaging control method characterized by the above.

17. A program for operating a computer as the imaging control device according to any one of claims 1 to 15.

Images