Imaging control device, imaging control method, and program

Abstract

To optimize exposure conditions while suppressing reduction in subject gradation.SOLUTION: An imaging control device includes acquisition means for acquiring distance information to a subject in a divided area obtained by dividing an imaging area, creating means for creating an integrated area by integrating the divided area on the basis of the distance information acquired by the acquisition means and a location of the divided area within an imaging area, and determining means for determining exposure conditions for the integrated area created by the creating means.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging control device, an imaging control method, and a program.

Background Art

[0002] There is known a technique of determining an AE target area for obtaining the brightness of a predetermined target image portion detected from a subject image, and determining the exposure amount of a solid-state imaging device based on image data representing an image within the determined AE target area (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to optimize exposure conditions while suppressing a decrease in the gradation of a subject.

Means for Solving the Problems

[0005] An imaging control device according to one aspect of the present invention includes acquisition means for acquiring distance information to a subject included in a divided area obtained by dividing an imaging area, and based on the distance information acquired by the acquisition means, A plurality of divided area each of creation means for creating an integrated area obtained by integrating the divided areas, 、 the at least two of the plurality of divided regions determining means for determining exposure conditions of the integrated area, Based on the brightness of the region corresponding to the integration region in the image captured in the imaging region, and is characterized by the above. comprising, the creating means creates the integration region by integrating at least two adjacent divided regions among the plurality of divided regions, in which the difference in the distance information is equal to or less than a first threshold value, and the exposure condition of the integration region determined by the determining means is uniformly applied to the at least two divided regions integrated into the integration region

Effects of the Invention

[0006] According to one aspect of the present invention, it is possible to optimize exposure conditions while suppressing a decrease in the gradation of a subject.​

Brief Description of the Drawings

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Mode for Carrying Out the Invention

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention, and not all combinations of the features described in the embodiments are essential for the solution means of the present invention. The configuration of the embodiments can be appropriately modified or changed according to the specifications of the device to which the present invention is applied and various conditions (usage conditions, usage environments, etc.). The technical scope of the present invention is determined by the scope of the claims and is not limited by the following individual embodiments.

[0009] <First Embodiment> FIG. 1 is a block diagram showing a configuration example of an imaging device according to an embodiment. Among the functional modules of the imaging device 100 shown in FIG. 1, for the functions realized by software, a program for providing the functions of each functional module is stored in a memory such as a ROM (Read Only Memory). Then, the program is read into a RAM (Random Access Memory) and executed by a CPU (Central Processing Unit) to be realized. For the functions realized by hardware, for example, by using a predetermined compiler, a dedicated circuit may be automatically generated on an FPGA from a program for realizing the functions of each functional module. FPGA is an abbreviation for Field Programmable Gate Array. Also, a gate array circuit may be formed in the same manner as the FPGA and realized as hardware. Also, it may be realized by an ASIC (Application Specific Integrated Circuit). Note that the configuration of the functional blocks shown in FIG. 1 is an example, and a plurality of functional blocks may constitute one functional block, or any one functional block may be divided into blocks that perform a plurality of functions.

[0010] In FIG. 1, the imaging device 100 can set exposure conditions such as shutter speed and analog gain for each divided area obtained by dividing the imaging area, and control the exposure conditions for each divided area to capture an image. The divided area may be composed of a single pixel or a plurality of pixels. When the divided area is composed of several pixels, the imaging area may be divided into blocks. The imaging device 100 may be used alone or mounted on a smartphone or a monitoring device or the like.

[0011] Note that in the following description, a single pixel or a plurality of pixels controlled under the same exposure conditions (exposure time and analog gain) may be referred to as a pixel area. The pixel area can be configured by a divided area obtained by dividing the imaging area. The number of pixels in each pixel area may be different. Also, in order to reduce the load of image processing, the imaging device 100 may collectively control a plurality of exposure areas having the same exposure conditions.

[0012] The imaging device 100 includes an imaging unit 101, an A / D (Analog / Digital) conversion unit 102, a signal processing unit 103, a D / A (Digital / Analog) conversion unit 104, an encoder unit 105, and a media I / F (Interface) unit 106. Further, the imaging device 100 includes a CPU 107, a ROM 108, and a RAM 109. Furthermore, the imaging device 100 includes an imaging system control unit 110, an operation unit 111, a character generation unit 112, a display unit 113, a distance information acquisition unit 114, a distance block creation unit 115, and an exposure condition determination unit 116.

[0013] The imaging unit 101 detects light from the subject for each pixel. The imaging unit 101 includes, for example, a zoom lens, a focus lens, an anti-shake lens, a diaphragm, a shutter, an optical low-pass filter, an IR (Infrared Rays) cut filter, a color filter, and an imaging sensor. The imaging sensor is, for example, a CMOS (Complementary Metal Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor.

[0014] The A / D conversion unit 102 converts the detected amount of light from the subject into a digital value. The signal processing unit 103 performs signal processing on the digital value output from the A / D conversion unit 102 to generate a digital image. The signal processing performed by the signal processing unit 103 includes, for example, demosaicking processing, white balance processing, and gamma processing. The D / A conversion unit 104 performs analog conversion on the digital image generated by the signal processing unit 103 for display.

[0015] The encoder unit 105 performs data compression processing on the digital image generated by the signal processing unit 103. For example, the encoder unit 105 performs processing such as data compression in the JPEG (Joint Photographic Experts Group) format. The media I / F unit 106 is an interface that connects the imaging device 100 to the media 117. The media 117 is, for example, a hard disk, a memory card, a CF (CompactFlash) card, an SD card, or a USB (Universal Serial Bus) memory, etc. The media 117 may be a PC (Personal Computer).

[0016] The CPU 107 performs the processing of the entire imaging device 100. The ROM 108 and the RAM 109 provide the CPU 107 with programs, data, working areas, etc. necessary for the processing. Also, when a control program necessary for the processing described later is stored in the ROM 108, it is once read into the RAM 109 and then executed by the CPU 107. Note that the CPU 107 may be a GPU (Graphics Processing Unit). The CPU 107 may be a single-core processor or a multi-core processor. The CPU 107 may have a function that operates as a neural network.

[0017] The operation unit 111 inputs instructions from the user. The operation unit 111 is, for example, a button, a mode dial, a touch panel attached to the display unit 113, etc. The character generation unit 112 generates numbers, characters, symbols, graphics, etc. to be displayed on the display unit 113. The display unit 113 displays images such as captured images and GUI (Graphical User Interface). The display unit 113 is, for example, a CRT (Cathode-Ray Tube), a liquid crystal display, or a touch screen. Input by the touch screen can also be treated as input of the operation unit 111.

[0018] The distance information acquisition unit 114 acquires distance information to the subject included in the divided area obtained by dividing the imaging area. Further, the distance information acquisition unit 114 transmits the acquired distance information to the distance distribution calculation unit 115. At this time, the distance information acquisition unit 114 may include a distance information calculation unit that calculates the distance from the imaging unit 101 to the subject. The distance information can be calculated based on, for example, at least any one of image plane phase difference AF (Auto Focus) using an imaging optical system, contrast AF using an imaging optical system, distance measurement detection using light detection, identification of a subject by image recognition, and background difference.

[0019] For example, when calculating the distance using image plane phase difference AF, one or more pixels for image plane phase difference AF are arranged in each pixel area in the image sensor. Thereby, the distance information acquisition unit 114 can calculate the distance information to the subject for each pixel area. Further, in image plane phase difference AF, since the distance to the subject can be calculated from the phase difference appearing on the image plane, it is possible to calculate the distance information for each pixel area and for each frame.

[0020] Further, the distance information acquisition unit 114 may acquire an evaluation value for each pixel area by contrast AF and estimate the distance to the subject from the position where the focus is achieved. In addition, a distance measurement technique using light detection typified by LIDAR (Light Detection and Ranging) may be used. Furthermore, the distance information acquisition unit 114 may identify the distance of the subject by image recognition. At this time, the distance information acquisition unit 114 estimates the distance for each pixel area from the size within the imaging angle of view of the subject. For example, for a specific subject such as the size of a person or a car, the size is determined to a certain extent. When using the subject identification means, the distance information acquisition unit 114 estimates the distance to the subject based on the information of the subject whose size is determined to a certain extent. In addition, when there is a pixel area that cannot be detected by these distance information detection means, the distance information acquisition unit 114 predicts based on the distance information of the peripheral area of the corresponding pixel area, for example, by the average value of the peripheral area.

[0021] Also, the user may set it arbitrarily, such as inputting distance information for each pixel region. Further, the distance information acquisition unit 114 may perform foreground / background segmentation using image recognition and determine the relative distance between the foreground and the background. In this case, as a distance measurement means, a special distance measurement mechanism such as pixels for image plane phase difference AF becomes unnecessary. Also, in the foreground / background segmentation, the background difference technique may be used. This technique performs determination based on the temporal change in brightness of the same pixel, rather than the difference in brightness between pixels at the same time. Therefore, the distance information acquisition unit 114 can acquire distance information even in a state where the brightness gradation is impaired.

[0022] Also, when the distance information acquisition unit 114 does not have a mechanism to acquire distance information for every pixel region, it may perform a complementation process such as linear complementation from the change amount of distance information at a plurality of image positions. Alternatively, the distance information acquisition unit 114 may measure the distance by grouping a plurality of pixel regions. However, in this case, the exposure condition determination unit 116 needs to set the exposure condition for each region obtained by grouping a plurality of pixel regions. Also, when there are a plurality of pixels for measuring distance within a pixel region, the distance information acquisition unit 114 may calculate the average value, median value, or the most frequent value of the histogram from the obtained plurality of distance data and determine it as the distance information of the corresponding pixel region.

[0023] The distance block creation unit 115 creates a distance block based on the distance information acquired from the distance information acquisition unit 114 and the position of the pixel region including the subject corresponding to the distance information. The distance block may include an integrated region obtained by integrating pixel regions based on the distance information acquired from the distance information acquisition unit 114 and the position of the pixel region including the subject corresponding to the distance information. The shape of the distance block is not limited to a square or a rectangle and may be a polygon.

[0024] At this time, the distance block creation unit 115 can calculate the relative distance between pixel regions from the distance information for each pixel region, and create distance blocks based on the relative distance. For example, if the relative distance to the subject among a plurality of pixel regions is close, the distance block creation unit 115 determines that those pixel regions are imaging a nearby subject (region), and determines that they are the same distance block. At this time, the distance block creation unit 115 may set a threshold value as a criterion for determining whether the relative distance is close or far, and if it is below the threshold value, it may be determined as the same distance block. This threshold value is preferably set by the designer or user based on the distance information of the entire image (maximum value, minimum value, average value, or median value of the distance), the imaging angle of view, the focal length, and the size constraints (minimum size and upper limit number) of the distance blocks. Thereby, it is possible to prevent the number of distance blocks from becoming extremely small, resulting in a decrease in harmony, or the number of distance blocks from becoming extremely large, increasing the processing load on the CUP 107. Further, the distance block creation unit 115 may create distance blocks based on the relative distance between the foreground and background segmented using image recognition.

[0025] It is desirable that the distance blocks are composed only of pixel regions adjacent to each other. Thereby, when a plurality of different subjects are located nearby within a range where they do not overlap (contact) in the imaging angle of view, the distance block creation unit 115 can determine them as different subjects and perform the process of creating distance blocks. However, when it is desired to process a plurality of different nearby subjects together, it is desirable that the distance block creation unit 115 is not limited to being composed only of adjacent pixel regions. However, it is desirable that the distance block creation unit 115 determines that they are also located nearby in the three-dimensional space and regards them as the same distance block only when the positional relationship (pixel distance between pixel regions) of the pixel regions in the image is close. At this time, the distance block creation unit 115 can set a threshold value for the positional relationship between two or more pixel regions in the image, and if it is below the threshold value, it can be determined as the same distance block.

[0026] The exposure condition determination unit 116 determines the exposure conditions for the distance blocks created by the distance block creation unit 115. At this time, the exposure condition determination unit 116 may limit the setting range of the exposure conditions for the distance blocks. For example, when the relative distance of the subject in the distance block is short, the exposure condition determination unit 116 may make the difference in the set values of the exposure conditions between adjacent pixel regions smaller than when the relative distance is long. Alternatively, the exposure condition determination unit 116 may determine the same exposure condition value for all pixel regions within the same distance block. Alternatively, when the difference in brightness within the distance block is equal to or less than the threshold value, the exposure condition determination unit 116 may narrow the setting range of the exposure conditions within the same distance block. Alternatively, the exposure condition determination unit 116 may limit the setting range of the exposure conditions only for the distance blocks where image recognition is performed or the distance blocks where the object of image recognition is detected.

[0027] The imaging system control unit 110 controls the imaging system as instructed by the CPU 107. The imaging system control unit 110 sets exposure conditions such as the shutter speed and analog gain for each pixel or region for the imaging unit 101. At this time, the imaging system control unit 110 can apply the exposure conditions determined by the exposure condition determination unit 116 to the imaging unit 101. In addition, the imaging system control unit 110 also performs controls such as focus adjustment, shutter opening and closing, and aperture adjustment. Note that the imaging device 100 may include various components other than the above-described components, but the description thereof is omitted.

[0028] At this time, the distance block creation unit 115 can acquire the distance information of the subject for each area within the imaging angle of view, and can create distance blocks by grouping areas that are relatively close to the subject. Then, the exposure condition determination unit 116 can set the same exposure conditions within the distance block, and can cause the imaging unit 101 to perform imaging via the imaging system control unit 110 so that the exposure conditions are reflected. As a result, the exposure condition determination unit 116 can cause imaging to be performed under different exposure conditions for each distance block in the entire image, improving the dynamic range of the entire image and suppressing a decrease in gradation in the captured image within the distance block.

[0029] For example, since the imaging device 100 can set exposure conditions for each area, the dynamic range of the captured image can be expanded by increasing the exposure amount in a dark area to perform imaging brighter and decreasing the exposure amount in a bright area to perform imaging darker. At this time, when setting exposure conditions for each area based only on the brightness of the area, while the merit of expanding the dynamic range can be obtained, it causes a decrease in contrast. Therefore, for a subject where clarification of contrast is important, visual or mechanical recognition becomes difficult, and there is a possibility of misrecognition. Here, subjects where clarification of contrast is important, such as the blue / yellow / red of a traffic signal, the black-and-white pattern of a crosswalk, and the characters on a sign, basically exist grouped together at relatively close positions.

[0030] The distance block creation unit 115 creates distance blocks by grouping regions that are relatively close to the subject, and the exposure condition determination unit 116 can set the exposure conditions within the distance blocks to the same settings. Therefore, the exposure condition determination unit 116 can limit the exposure conditions within the same distance block and image subjects with a relatively close three-dimensional relative distance under close exposure conditions. As a result, even when the subject has depth and contrast clarification is important, the exposure condition determination unit 116 can capture an image that maintains the contrast of the subject and improve the recognizability. For example, the distance block creation unit 115 determines that the blue / yellow / red regions of the traffic signal are the same distance block, and the exposure condition determination unit 116 can image the blue / yellow / red regions of the traffic signal under close exposure conditions and maintain the contrast. Also, the distance block creation unit 115 determines that the black-and-white pattern of the crosswalk is the same distance block, and the exposure condition determination unit 116 can image the black-and-white pattern of the crosswalk under close exposure conditions and maintain the contrast.

[0031] As described above, according to the above-described embodiment, the imaging device 100 determines the exposure conditions of the integrated region obtained by integrating the divided regions based on the distance information to the subject included in the divided regions obtained by dividing the imaging region and the positions of the divided regions within the imaging region. Thereby, the imaging device 100 can optimize the exposure conditions while suppressing a decrease in the tonal harmony of the subject.

[0032] In the example of FIG. 1, the distance information acquisition unit 114, the distance block creation unit 115, and the exposure condition determination unit 116 are shown as blocks separate from the CPU 107. The imaging control process executed by the distance information acquisition unit 114, the distance block creation unit 115, and the exposure condition determination unit 116 may be executed by the CPU 107 as a part of the functions realized by the CPU 107.

[0033] FIG. 2 is a diagram showing an example of a captured image according to an embodiment. In the description of this embodiment, since the pixel region is composed of a small number of pixels and is sufficiently fine with respect to the image size, illustration of the pixel region is omitted. However, if there are at least two pixel regions in the image, the present invention is applicable.

[0034] In FIG. 2, in the captured image 200, as main subjects, there are vehicles 201, 202, a road 203, sidewalks 204, 205, a crosswalk 206, and a white line 207 of the road. The vehicle 201 and the sidewalk 204 are at positions close to the imaging surface. The vehicle 202 and the sidewalk 205 are at positions far from the imaging surface. The crosswalk 206 exists from a position close to the imaging surface to a position far from the imaging surface and has depth.

[0035] At this time, the distance block creation unit 115 compares the distance information between the pixel regions including the subjects and calculates the relative distance of the subjects included in the pixel regions. Further, the distance block creation unit 115 sets a threshold for this relative distance, and if the relative distance is equal to or less than the threshold, it determines that the region is a region where the distance of the subject is close and determines the same distance block.

[0036] In the two-dimensional plane in the captured image 200, the vehicle 201 is in contact with the road 203 and the white line 207 of the road, but is a subject having a height with respect to the road 203 and the white line 207 of the road. Therefore, when viewed as the distance from the imaging surface to the subject, the vehicle 201 has a relative distance equal to or greater than the threshold with respect to the road 203 and the white line 207 of the road, and the distance block creation unit 115 determines that they are different distance blocks. However, since the vehicle 201 and the road 203 are in contact with the ground at the tire portion of the vehicle 201, the relative distance is small for this portion. Therefore, the distance block creation unit 115 cannot determine that different subjects are different distance blocks only based on the simple relative distance between the subjects. Therefore, it is desirable for the distance block creation unit 115 to determine the ground contact surface of the vehicle 201 and divide the distance blocks with the ground contact surface as a boundary. The method for determining the ground contact surface will be described later.

[0037] Similarly, the vehicle 202 is a subject that is at a height relative to the road 203 and the white line 207 of the road, and the distance block creation unit 115 determines that they are different distance blocks.

[0038] Also, the lower side of the image of the road 203 is relatively close in distance, and the upper side of the image is relatively far in distance. Even for a subject whose distance to the subject varies depending on the position within the image, if the distance to the subject between pixel regions changes smoothly without a sharp change, the relative distance to the subject between pixel regions becomes close, and they are regarded as the same distance block. At this time, the distance block creation unit 115 determines that the relative distances are close in the road 203, the sidewalks 204, 205, the crosswalk 206, and the white line 207 of the road, and that they are the same distance block. The distance block creation unit 115 creates the distance blocks in FIG. 3 based on these determination results.

[0039] FIG. 3 is a diagram showing an example of creating distance blocks for the captured image in FIG. 2. In FIG. 3, the distance block creation unit 115 creates a distance block map 300 based on, for example, the captured image 200 in FIG. 2. The distance block map 300 includes distance blocks 301 to 303.

[0040] The distance block 301 is the pixel region in which the vehicle 201 in FIG. 2 was captured, and is illustrated by black filling. The distance block 302 is the pixel region in which the vehicle 202 in FIG. 2 was captured, and is illustrated by black dots. The distance block 303 is the captured region of the road 203, the sidewalks 204, 205, the crosswalk 206, and the white line 207 of the road in FIG. 2, and is illustrated by white filling. In this way, the distance block creation unit 115 can create three distance blocks 301 to 303 from the captured image 200 in FIG. 2.

[0041] Here, since the imaging device 100 can set exposure conditions for each region, it is possible to increase the exposure amount in a dark region to capture an image brighter, and decrease the exposure amount in a bright region to capture an image darker, thereby expanding the dynamic range of the captured image. However, if the exposure conditions are set so that the dynamic range of the crosswalk 206 in FIG. 2 is expanded, the brightness gradation of the crosswalk 206, where the gradation of brightness between white and black is important, will be impaired.

[0042] For such a subject, if the brightness gradation is impaired, it becomes difficult to distinguish between white and black of the crosswalk 206, and the accuracy of image recognition decreases. Also, in a scene where such a difference in brightness is compared, since the brightness changes depending on the light source (environment), it is assumed that there is black and white in the nearby crosswalk 206.

[0043] Here, the distance block creation unit 115 can determine that a subject, such as the crosswalk 206 in FIG. 2, where the gradation of brightness between white and black is important, is the same distance block 303. Then, the exposure condition determination unit 116 can suppress a decrease in gradation by causing imaging to be performed under the same exposure conditions for the pixel regions within the same distance block. As a result, the exposure condition determination unit 116 can obtain an image in which the difference in brightness between black and white is maintained in the imaging region of the crosswalk 206 while being able to expand the dynamic range over the entire captured image 200.

[0044] Here, in the captured image 200, the distance to the subject on the lower side of the image is small, and the distance to the subject on the upper side of the image is large. Therefore, the distance from the imaging device 100 to the road 203 changes according to the position within the image. Accordingly, in order for the distance block creation unit 115 to determine that the entire area of the road 203 is the same distance block 303, instead of the absolute value of the distance for each pixel region, the relative distances for each pixel region are compared, and regions with close relative distances are determined to be continuous subjects (subjects with depth). However, the distance block creation unit 115 may create distance blocks using absolute values, such as when imaging a scene with little depth.

[0045] Also, in the example of FIG. 3, the distance block creation unit 115 determined three distance blocks 301 to 303, but the number of distance blocks varies depending on the threshold value of the relative distance. If the distance information acquired by the distance information acquisition unit 114 has sufficient accuracy, the distance block creation unit 115 can increase the number of distance blocks by relaxing the threshold value. For example, by relaxing the threshold value, the distance block creation unit 115 can identify the road 203, the sidewalks 204 and 205, the crosswalk 206, and the white line 207 of the road as different distance blocks. As a result, the exposure condition determination unit 116 can set the exposure condition for each subdivided distance block, and can improve the high definition of the harmony for each subject while expanding the dynamic range. Also, by restricting the threshold value, the number of distance blocks may decrease. In this case, the exposure condition determination unit 116 can reduce the processing load for setting the exposure condition.

[0046] FIG. 4 is a flowchart showing the imaging process according to the embodiment. Note that each step in FIG. 4 is realized by the distance block creation unit 115 and the exposure condition determination unit 116 reading and executing the program stored in the storage unit of the imaging device 100 in FIG. 1. Also, at least a part of the flowchart shown in FIG. 4 may be realized by hardware. When realized by hardware, for example, a dedicated circuit may be automatically generated on the FPGA from the program for realizing each step by using a predetermined compiler. Also, a Gate Array circuit may be formed in the same manner as the FPGA and realized as hardware. Also, it may be realized by an ASIC. In this case, each block in the flowchart shown in FIG. 4 can be regarded as a hardware block. Note that a plurality of blocks may be combined and configured as one hardware block, or one block may be configured as a plurality of hardware blocks.

[0047] In S401 of FIG. 4, the distance information acquisition unit 114 acquires the distance information of the subject for each pixel region from the distance information acquisition unit 114. Next, in S402 of FIG. 4, the distance block creation unit 115 creates a distance block based on the distance information acquired from the distance information acquisition unit 114 and the position of the pixel region including the subject corresponding to the distance information.

[0048] Next, in S403 of FIG. 4, the exposure condition determination unit 116 sets the exposure conditions to the same value for each distance block for all pixel regions within the same distance block. At this time, the exposure condition determination unit 116 may determine the exposure conditions that result in proper exposure based on the brightness within the distance block of the captured image and the exposure conditions at the time of imaging so that proper exposure is achieved for each distance block. In proper exposure, there are few data losses such as black crush and white clipping in the image according to the brightness of the subject, and an image with high visibility captured at an appropriate brightness can be obtained. Next, in S404 of FIG. 4, the imaging system control unit 110 causes the imaging unit 101 to perform imaging based on the exposure conditions set in S403.

[0049] Next, in S405 of FIG. 4, the encoder unit 105 performs image processing such as compression for each distance block. As a result, in the imaging device 100, an image is captured with proper exposure for each distance block, and the amount of compression is minimized for each distance block. Therefore, the imaging device 100 can acquire an image in which the reduction in brightness gradation is suppressed for each distance block. Also, when viewing the entire image, the imaging device 100 can set different exposure conditions between distance blocks, so an image with an improved dynamic range can be acquired. Therefore, by setting the exposure conditions for each distance block, the imaging device 100 can acquire an image in which the reduction in brightness gradation is suppressed regardless of the type and number of subjects while expanding the dynamic range. Note that the type of subject may be not only the foreground and background but also another object such as a person, a vehicle, or a road.

[0050] In the case of the imaging device 100 capable of setting exposure conditions for each pixel region, by performing imaging under different exposure conditions for each pixel region, it becomes possible to acquire an image with an extended dynamic range. On the other hand, although the range (data amount) of brightness is widened, due to the constraints on the amount of data to be displayed as image data and the capabilities of the CPU 107 that performs image processing, data compression is required. Therefore, the compression rate increases by the amount corresponding to the increase in the range (data amount) of brightness, and the gradation of that brightness decreases.

[0051] For example, when using an image sensor capable of expressing a single exposure amount in 14 bits, assume a scene where two regions, the darkest region (14 bits) and the brightest region (14 bits), are imaged simultaneously. At this time, if the value that can be imaged brightest in the dark region (maximum value) and the value that can be imaged darkest in the bright region (minimum value) do not overlap by 1 bit, the image sensor will image in 28 bits (14 bits + 14 bits), and the gradation of brightness will not be impaired.

[0052] However, in reality, due to constraints such as the final output data having a fixed format such as JPEG (8 bits), data compression is performed. Considering compression to 8 bits, 28-bit data is compressed to 8 bits. At this time, the imaging device 100 can set all the exposure conditions within the same distance region to be the same, and it is only necessary to compress 14-bit data to 8 bits, and the gradation degradation due to data compression can be suppressed.

[0053] Hereinafter, a supplementary explanation of the ground plane determination method will be given with reference to FIGS. 2 and 3. Since the ground contact surfaces of the vehicle 201 and the road 203 are close to the subject in terms of the relative distance to the subject, the distance block creation unit 115 may not be able to determine them as different distance blocks by referring only to the relative distance. Even when the distance block creation unit 115 determines the subject as having different distances in this way, if there is a ground contact surface, the distance blocks cannot be segmented by referring only to the relative distance. Therefore, it is desirable for the distance block creation unit 115 to set a method for determining distance blocks in addition to referring to the relative distance. At this time, it is desirable for the distance block creation unit 115 to include a determination means for determining the boundary of the distance block. Further, it is desirable for the distance block creation unit 115 to segment the distance blocks with the position in the image determined as the ground contact surface as the boundary.

[0054] For example, in the relationship of the relative distances between the vehicle 201 and the road 203 in FIG. 2, at the highest part (roof) of the vehicle 201, the relative distance to the road 203 is the largest. On the other hand, at the lowest part (tire) of the vehicle 201, the relative distance to the road 203 is the smallest, and further becomes zero at the ground contact surface. In this way, the relative distance between the vehicle 201 and the road 203 gradually changes in the distance block creation unit 115, and it is necessary to determine different distance blocks at positions where the relative distances are close. Therefore, the distance block creation unit 115 sets a threshold for the change in the relative distance with respect to the boundary between such subjects with changing relative distances, and determines the position where it is below the threshold as the ground contact surface. Also, the distance block creation unit 115 may determine the point where the relative distance is the smallest as the ground contact surface for the pixel region where the relative distance changes.

[0055] Also, the distance block creation unit 115 may use image recognition such as edge detection in combination with the distance information for the identification of the ground contact surface. When a specific subject (for example, the vehicle 201 in FIG. 2) can be identified by image recognition, the distance block creation unit 115 segments the distance blocks at the boundary between the specific subject and the background (the road 203 in FIG. 2). Thereby, the distance block creation unit 115 can appropriately create distance blocks even when there is a ground contact surface.

[0056] Alternatively, the distance block creation unit 115 may convert a two-dimensional image into a three-dimensional model using image processing, calculate the ground plane, and create a distance block. In this case, it is desirable that the distance block creation unit 115 generates a three-dimensional model from a plurality of viewpoints using images captured from a plurality of imaging devices or a plurality of positions, and identifies the ground plane. Further, the distance block creation unit 115 may compare the brightness changes within a single image of the subject, and use it as an aid in determining the ground plane on the assumption that a point with a brightness change is highly likely to be the ground plane of the subject.

[0057] Also, the orientation of the ground plane is not limited to the horizontal direction, and may be the vertical direction, the diagonal direction, or the like. For example, in the imaging angle of view of FIG. 2, when it is determined that the ground is the ground plane, the distance block creation unit 115 can determine that there is a ground plane below the distance block. On the other hand, when the imaging device 100 is turned upside down, the captured image is inverted. Therefore, the distance block creation unit 115 can determine that there is a ground plane at the upper boundary of the distance block with respect to the captured image.

[0058] As described above, when the position of the ground plane with respect to the distance block is known, the distance block creation unit 115 can improve the creation accuracy of the distance block by setting the ground plane in advance. Also, the boundary between subjects is not limited to the ground, and also occurs due to contact between subjects. Therefore, it is desirable to determine the direction of the boundary between subjects with respect to the image not only in the horizontal and vertical directions but also in the diagonal direction.

[0059] Note that in S403 of FIG. 4, the exposure condition determination unit 116 sets the exposure conditions to the same value for all pixel regions within the same distance block, but they do not necessarily have to be the same exposure conditions. The exposure condition determination unit 116 may set the exposure conditions for each pixel region within the distance block after restricting the range in which the exposure conditions can be set within the distance block.

[0060] When the exposure condition determination unit 116 sets different exposure conditions within the distance block, the range of brightness that can be imaged expands, the data volume is expanded, and the dynamic range within the distance block can be improved. However, the exposure condition determination unit 116 needs to limit the range in which the exposure conditions can be set in order to suppress the amount of data compression. The narrower the range in which the exposure conditions can be set, the better the brightness gradation within the same distance block, but the lower the dynamic range. Conversely, the wider the range in which the exposure conditions can be set, the lower the brightness gradation within the same distance block, but the higher the dynamic range. Therefore, the exposure condition determination unit 116 compares the brightness within the distance block for each pixel region, and when the difference in brightness is large, it is desirable to give priority to the dynamic range and widen the range in which the exposure conditions can be set. However, the exposure condition determination unit 116 limits the exposure conditions that can be set within at least one distance block to be less than the maximum range that can be set for the imaging device 100. Thereby, the exposure condition determination unit 116 can suppress a decrease in brightness gradation.

[0061] Also, the exposure condition determination unit 116 desirably makes the difference in the set values of the exposure conditions smaller as the relative distance to the subject between the pixel regions is closer. Thereby, it is possible to perform imaging with suppressed gradation degradation for a subject closer where the difference in brightness is important.

[0062] Also, the exposure condition determination unit 116 determines whether the difference in brightness is small by setting a threshold for the difference in brightness within the same distance block. At this time, since the smaller the difference in brightness within the distance block, the less the dynamic range is required, the exposure condition determination unit 116 can further prevent a decrease in gradation by narrowing the range in which the exposure conditions can be set.

[0063] In addition, the imaging device 100 does not necessarily always perform the processing related to the distance block in FIG. 4 (S401, S402, or S403). The imaging device 100 may start the processing related to the distance block based on at least any one of the detection accuracy of image recognition, the detection result of image recognition, the range of the brightness of the entire image, and the difference in distance of the pixel regions in the entire image. For example, the imaging device 100 may start the processing related to the distance block starting from the low accuracy of image recognition or the fact that the accuracy of image recognition has become low. Thereby, when the accuracy of image recognition is sufficient, the processing in FIG. 4 becomes unnecessary, and the load on the imaging device 100 can be reduced. In addition, the imaging device 100 may start the processing related to the distance block starting from the fact that what could be recognized by image recognition can no longer be recognized.

[0064] In addition, the imaging device 100 may start the processing related to the distance block starting from the fact that the range of the brightness of the entire image is large and a decrease in gradation is expected. At this time, the imaging device 100 can set a threshold value for the range of the brightness of the entire image and start the processing related to the distance block when the brightness range is equal to or greater than the threshold value. In addition, when the difference in distance of the pixel regions in the entire image is small, since the distance block is not created, there is no need to perform the processing related to the distance block. Therefore, the imaging device 100 can reduce the processing load by setting a threshold value for the difference in distance of the pixel regions in the entire image and performing the processing related to the distance block when the difference in distance becomes large.

[0065] In the present embodiment, the imaging device 100 can image all pixel regions at the same timing (one frame). Therefore, no deviation occurs in the imaging timing between the regions. That is, since it is possible to image an image at the same timing, even when the exposure times are different, it is possible to reduce the omission of the subject and the occurrence of double images as compared with the case of imaging a plurality of images for each pixel region. In addition, the imaging device 100 does not necessarily have to perform the processing in FIG. 4 for each frame. The imaging device 100 may perform the processing in FIG. 4 for every several frames.

[0066] In addition, although the exposure condition determination unit 116 may set the exposure conditions within the distance block to proper exposure, it is not necessarily required to set them to proper exposure. It may be set to underexposure or overexposure, or may be manually set.

[0067] In addition, the distance block creation unit 115 may determine that distance blocks with a close block distance are located close to each other in the three-dimensional space and determine them as the same distance block. In the present embodiment, the block distance of the distance block is defined as the distance to the subject imaged by the distance block. When there is a plurality of distance information within the same distance block, the block distance can be defined by the median value, average value, mode value, maximum value, or minimum value of these distance information. The exposure condition determination unit 116 can apply the same exposure conditions to distance blocks with a close block distance. At this time, the exposure condition determination unit 116 can set a threshold value for the difference in the block distance between the distance blocks, and if it is below the threshold value, it can be determined that the distance blocks (subjects) are close to each other.

[0068] As a result, the imaging device 100 can image a plurality of different subjects located close to each other under the same exposure conditions, and the brightness harmony between the plurality of different subjects can also be improved. At this time, the distance block creation unit 115 may refer to not only the block distance of the distance block but also the positional relationship of the distance blocks in the image (the distance between the distance blocks). Thereby, the distance block creation unit 115 can predict that if the distance blocks are close to each other in the image, they are also close to each other in reality, similar to when referring to the positional relationship of the pixel regions in the image. Therefore, the distance block creation unit 115 may perform processing by treating distance blocks that are close to each other in the image as the same distance block.

[0069] The exposure condition determination unit 116 may limit the exposure conditions between adjacent distance blocks. For example, when the brightness of the images of the distance blocks is similar between adjacent distance blocks, the exposure condition determination unit 116 can widen the difference in exposure conditions to widen the brightness within the image between the distance blocks, making it easier to recognize the boundaries of the distance blocks from the image. Conversely, when the brightness of the images of the distance blocks is similar between adjacent distance blocks, the exposure condition determination unit 116 may narrow the difference in exposure conditions. In that case, the exposure condition determination unit 116 can reduce the brightness step due to the exposure conditions, making it difficult to recognize the boundaries of the distance blocks and enabling a natural image to be captured. At this time, the exposure condition determination unit 116 sets a threshold for the difference in brightness between the distance blocks within the image and determines whether the brightness between the distance blocks within the image is equal to or greater than the threshold, or equal to or less than the threshold.

[0070] Note that the exposure condition determination unit 116 does not necessarily have to set the same exposure conditions for all distance blocks. The exposure condition determination unit 116 may set the same exposure conditions for at least one distance block. For example, when the area for image recognition is known in advance or when the object to be recognized is detected, the exposure condition determination unit 116 can reduce the processing load by setting the same exposure conditions only for the corresponding distance blocks. Also, the exposure condition determination unit 116 can suppress a reduction in harmony for all areas within the image by setting the same exposure conditions for each distance block for all distance blocks. Note that the user may specify the distance blocks for which the exposure conditions are to be restricted.

[0071] <Other Embodiments> The present invention may supply a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium. Then, one or more functions of the above-described embodiments can also be realized by processing in which one or more processors in the computer of the system or device read and execute the program. Also, they can be realized by a circuit (for example, FPGA or ASIC) that realizes one or more functions.

Description of Reference Numerals

[0072] 100 Imaging device, 101 Imaging unit, 102 A / D conversion unit, 103 Signal processing unit, 104 D / A conversion unit, 105 Encoder unit, 106 Media I / F unit, 107 CPU, 108 ROM, 109 RAM, 110 Imaging system control unit, 111 Operation unit, 112 Character generation unit, 113 Display unit, 114 Distance information acquisition unit, 115 Distance block creation unit, 116 Exposure condition determination unit

Claims

1. acquisition means for acquiring distance information to a subject included in each of a plurality of divided areas obtained by dividing an imaging area; creation means for creating an integrated area by integrating at least two of the plurality of divided areas based on the distance information acquired by the acquisition means; determination means for determining an exposure condition of the integrated area based on the brightness of an area corresponding to the integrated area in an image captured in the imaging area, the imaging control device comprising: the creation means creates the integrated area by integrating at least two adjacent divided areas among the plurality of divided areas, the difference in the distance information of which is equal to or less than a first threshold; the exposure condition of the integrated area determined by the determination means is uniformly applied to the at least two divided areas integrated into the integrated area.

2. The imaging control device according to claim 1, wherein the first threshold is determined based on at least one of distance information of the entire image, an imaging angle of view, a focal length, or a size constraint of the integrated area.

3. The imaging control device according to claim 1 or 2, wherein the acquisition means calculates distance information of the integrated area based on the distance information of the at least two divided areas integrated into the integrated area.

4. The imaging control device according to claim 3, wherein the creation means further integrates adjacent integrated areas, the difference in the distance information between which is equal to or less than a second threshold, into one integrated area.

5. The imaging control device according to claim 3, wherein the acquisition means calculates distance information of the integrated area based on a median value, an average value, a mode value, a maximum value, or a minimum value of the distance information of the at least two divided areas.

6. The imaging control device according to any one of claims 1 to 5, wherein the determination means restricts the setting range of the exposure conditions only to the integration area where image recognition is performed or the integration area in which the object of image recognition is detected.

7. The distance information of each of the divided areas is calculated based on at least one of image plane phase difference AF (Auto Focus) using an imaging optical system, contrast AF using an imaging optical system, distance measurement detection using light detection, identification of a subject by image recognition, and background difference, the imaging control device according to any one of claims 1 to 6.

8. The imaging control device according to any one of claims 1 to 7, wherein the determination means determines the exposure conditions of the integration area such that the difference in brightness of the images corresponding to the respective adjacent integration areas is equal to or less than an eighth threshold value.

9. The imaging control device according to any one of claims 1 to 7, wherein the determination means determines the exposure conditions of the integration area such that the difference in brightness of the images corresponding to the respective adjacent integration areas is equal to or greater than an eighth threshold value.

10. An acquisition step of acquiring distance information to a subject included in each of a plurality of divided areas obtained by dividing an imaging area; A creation step of creating an integration area by integrating at least two of the plurality of divided areas based on the acquired distance information; A determination step of determining the exposure conditions of the created integration area based on the brightness of the area corresponding to the integration area in the image captured in the imaging area, and comprising: In the creation step, the integration area is created by integrating at least two adjacent divided areas whose difference in distance information is equal to or less than a first threshold value among the plurality of divided areas; The exposure conditions of the integration area determined in the determination step are uniformly applied to the at least two divided areas integrated into the integration area, an imaging control method.

11. A program for causing a computer to operate as the imaging control device according to any one of claims 1 to 9.

Images