Imaging device, information processing device, control method, program, and storage medium

Abstract

To provide an image capturing apparatus configured to appropriately set exposure conditions for each of regions of an image capturing surface of an imaging element even in a case where an image luminance distribution changes with time due to movements of a subject, and to provide a control method of the apparatus, a program, an information processing apparatus, and a storage medium.SOLUTION: In an imaging system, an image capturing apparatus 101 includes: an image capturing unit 201 configured so that a parameter related to exposure can be set for each of regions of an image capturing surface; a detection unit 301 configured to detect a first region in a first image captured by the image capturing unit; a calculation unit 302 configured to calculate the exposure-related parameter based on a luminance of the first region in the first image; a prediction unit 303 configured to predict a second region captured after the first image, the second region being similar to the first region; and a control unit 304 configured to control the image capturing unit so that the exposure-related parameter calculated based on the luminance of the first region is applied to the second region predicted by the prediction unit to capture a second image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a control method, a program, and a storage medium. [Background technology]

[0002] Patent Document 1 discloses a technique for widening the dynamic range of an image sensor by setting exposure conditions for each of a plurality of regions on the imaging surface of the image sensor. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 5247397 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to set suitable exposure conditions for each of a plurality of regions on the imaging surface even when the luminance distribution of an image changes over time, such as when a subject moves. [Means for solving the problem]

[0005] In order to solve the above problem, an imaging device according to one aspect of the present invention includes an imaging unit having an imaging element capable of setting exposure parameters for each of multiple regions on an imaging surface; a detection unit that detects a first region in a first image captured by the imaging unit; a calculation unit that calculates the exposure parameters based on the luminance of the first region in the first image; a prediction unit that predicts a second region in a second image captured after the first image, the second region being similar to the first region; and a control unit that applies the exposure parameters calculated based on the luminance of the first region to the second region predicted by the prediction unit and controls the imaging unit to capture the second image. [Effects of the Invention]

[0006] According to the present invention, even when the luminance distribution of an image changes over time, such as when a subject moves, it is possible to set suitable exposure conditions for each of a plurality of regions on the imaging surface. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of an imaging system according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the device configuration of an imaging device according to a first embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the functional configuration of an imaging apparatus according to a first embodiment. [Figure 4] 3A and 3B are diagrams showing an example of an image captured by the imaging device according to the first embodiment and a histogram for each region. [Figure 5] 5 is a flowchart showing an example of the operation of the imaging apparatus according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing an example of the operation of the imaging device according to the first embodiment. [Figure 7] 10 is a flowchart showing an example of the operation of the imaging device according to a modified example of the first embodiment. [Figure 8] FIG. 10 is a diagram for explaining a method for determining a feature region according to the second embodiment. [Figure 9] FIG. 10 is a diagram for explaining whether adjacent exposure regions are continuous in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the accompanying drawings. The embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiments. Furthermore, a configuration may be made by appropriately combining parts of each embodiment described below.

[0009] <Embodiment 1> (System Configuration) 1 is a diagram showing an example of the configuration of an imaging system according to this embodiment. The imaging system 100 includes an imaging device 101, a network 102, an information processing device 103, a display device 104, and an input device 105.

[0010] The imaging device 101 is capable of communicating with an information processing device 103 via a network 102. The imaging device 101 captures an image of a subject, generates image data, and transmits the image data to the information processing device 103 via the network 102.

[0011] A display device 104 and an input device 105 are connected to the information processing device 103. Image data received by the information processing device 103 is output to the display device 104, and an image captured by the imaging device 103 is displayed on a display unit of the display device 104. The input device 105 is a keyboard, a mouse, or the like, and is an interface for inputting operation information for the information processing device 103 and the imaging device 101. The operation information is, for example, instructions on imaging conditions for the imaging device 101 and instructions for PTZ (Pan Tilt Zoom) operation.

[0012] In this embodiment, the client device 103, the display device 104, and the input device 105 are separate entities, but the information processing device 103, the display device 104, and the input device 105 may be integrated into one device, as in a notebook computer. Also, the imaging device 101 and the information processing device 103 do not need to be connected via the network 102, and may be directly connected to one another. Furthermore, the imaging device 101, the client device 103, the display device 104, and the input device 105 may all be integrated into one device, as in a consumer camera with a touch panel display.

[0013] (Device configuration) 2 is a diagram showing an example of the configuration of an imaging device according to this embodiment. The imaging device 101 includes an imaging unit 201, an encoder 202, a network I / F 203, a CPU (Central Processing Unit) 204, a RAM (Random Access Memory) 205, and a ROM (Read Only Memory) 206.

[0014] The imaging optical system 200 is a lens that focuses light from a subject onto the imaging surface of an imaging element 201a (described later), and is configured with, for example, a zoom lens, a focal lens, a blur correction lens, etc. In this embodiment, the imaging optical system 200 is separate from the imaging device 101 and is detachably provided to the imaging device 1010, but the imaging optical system 200 and the imaging device 101 may also be configured as an integrated unit.

[0015] The imaging unit 201 includes an imaging element 201a, an amplifier 201b, and an image processing unit 201c. The imaging element 201a, the amplifier 201b, and the image processing unit 201c may be separate or integrated. The imaging element 201a is, for example, a charge coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor. Pixels made of photoelectric conversion elements are arranged two-dimensionally on the imaging surface of the imaging element 201a, and light from a subject collected by the imaging optical system 200 is converted into an electrical signal and output. In addition, exposure parameters can be set for each of multiple regions made up of multiple pixels (for example, a total of 128 × 128 pixels). The exposure parameters include, for example, exposure time and gain.

[0016] The amplifier 201b amplifies and outputs the electrical signal output from the image sensor 201a. An amplifier 201b is provided for each pixel, and its signal amplification rate (analog gain) is included in exposure-related parameters and can be set for each of multiple regions of the image sensor 201a. Each region of the image sensor 201a is referred to as an exposure region or a divided region. In other words, each region representing a group of pixels on the imaging surface of the image sensor 201a, for which exposure-related parameters are independently controlled, is an exposure region.

[0017] The image processing unit 201c performs A / D conversion to convert the analog electrical signal output from the amplifier 201b into a digital signal. Furthermore, the image processing unit 201c performs image processing, including demosaicing, white balance processing, and gamma processing, on the converted digital data (image data). The image processing unit 201c also corrects the brightness of the image data by increasing or decreasing the digital value of the image data corresponding to each of the multiple regions of the image sensor 201a. The brightness correction value (digital gain) can also be set for each of the multiple regions, and is included in the exposure parameters.

[0018] The encoder 202 encodes the image data output from the imaging unit 201 into a file format such as Motion JPEG, H264, or H265, and outputs the encoded data.

[0019] The network I / F 203 is an interface that transmits image data output from the encoder 202 to the information processing device 103 via the network 102. In this embodiment, the encoded image data is transmitted to the information processing device 103, but it may also be stored in a removable external storage device such as an SD card or an internal storage device such as the ROM 206.

[0020] The CPU 204 is a central processing unit that controls the image capturing apparatus 101 .

[0021] The RAM 205 provides a work area used when the CPU 204 executes processing, and also functions as a frame memory and a buffer memory.

[0022] The ROM 206 stores programs and image data for the CPU 204 to control the image capturing apparatus 101 .

[0023] (Functional configuration) FIG. 3 is a diagram illustrating an example of the functional configuration of an imaging device according to this embodiment. For software-implemented functions of the functional blocks shown in FIG. 3, a program for providing the function of each functional block is stored in a storage medium such as the ROM 206. The program is then loaded into the RAM 205 and executed by the CPU 204. For hardware-implemented functions, a dedicated circuit may be automatically generated on an FPGA from a program for implementing the function of the functional block written using, for example, a predetermined compiler. FPGA stands for Field Programmable Gate Array. Alternatively, a gate array circuit may be formed in a similar manner to an FPGA to implement the function as hardware. Alternatively, the function may be implemented using an ASIC (Application Specific Integrated Circuit). The functional block configuration shown in FIG. 3 is merely an example; multiple functional blocks may constitute a single functional block, or one functional block may be divided into blocks performing multiple functions.

[0024] The imaging device 101 includes an imaging unit 201 , a calculation unit 301 , a detection unit 302 , a prediction unit 303 , a control unit 304 , and a communication unit 305 .

[0025] As described above, the imaging unit 201 includes an imaging element 201a that can set exposure-related parameters for each of multiple regions on the imaging surface. The exposure-related parameters include at least one of exposure time (shutter speed), analog gain, digital gain, and exposure value, and are also called exposure conditions.

[0026] The detection unit 301 detects a first region in a first image captured by the imaging unit 201. The first region refers to a subject region or a region that satisfies a predetermined condition. Specific examples of the predetermined condition are shown below. FIG. 4 is a diagram illustrating an example of a first image captured by the imaging unit 201 of the imaging device 101 according to this embodiment and a first region in the first image. Grid lines in the first image 401 indicate the boundaries of image regions corresponding to multiple regions on the imaging surface of the image sensor 201a. In this embodiment, an example is described in which the first region 402 corresponds to one region on the imaging surface of the image sensor 201a. However, multiple regions on the imaging surface of the image sensor 201a may correspond to the first region. As shown in FIG. 4, most of the first region 402 is a blown-out highlight region, and the remaining regions are also close to saturated. The graphs at the bottom of FIG. 4 show example luminance histograms for the first region. The histogram on the left corresponds to the first region 402. The histogram on the right is another example. The horizontal axis of the graph indicates the luminance gradation, and the vertical axis indicates the output frequency (number of pixels). These histograms are obtained for each exposure area and show the number of pixels outputting each gradation in each exposure area. The luminance histogram for the first area 402 shows that the output frequency of the saturated luminance TH_H is the highest, and the output frequencies of the other luminances are also shifted to the right. In this embodiment, the predetermined condition is that the output frequency of TH_H is equal to or greater than a predetermined threshold value Th_pix. This makes it possible to detect areas with many blown-out highlights as the first area. The histogram at the bottom right of Figure 4 shows an example in which the predetermined condition is that the output frequency of the luminance TH_L of a blocked-up shadow area is equal to or greater than a predetermined threshold value Th_pix. In this case, areas with many blocked-up shadow areas can be detected as the first area.

[0027] As described above, in this embodiment, a region in which the number of pixels having a predetermined brightness (pixel value) is equal to or greater than a predetermined threshold is detected as a characteristic region (first region). Note that an exposure region in which the number of pixels having a brightness equal to or less than the first threshold or the number of pixels having a brightness equal to or greater than the second threshold is equal to or greater than a third threshold may also be detected as a characteristic region. In other words, the detection unit 301 detects an exposure region that satisfies predetermined conditions as a characteristic region.

[0028] The method of detecting a characteristic region by the detection unit 301 is not limited to the above method, and the characteristic region may be detected by a known technique such as subject detection. As described above, the characteristic region detected by the detection unit 301 includes a subject region. In other words, a region of a moving subject detected by image analysis such as brightness change or background difference is also an example of a characteristic region, and may be detected by the detection unit 301. Note that the detection unit 301 may function as an extraction unit that extracts the feature amount of the subject in the characteristic region.

[0029] Furthermore, the detection unit 301 detects the area adjacent to the first area as the first area based on the result of comparing the exposure parameters of the area adjacent to the feature area (first area) with the exposure parameters of the feature area. More specifically, if the difference between the exposure conditions of the area adjacent to the feature area and the exposure conditions of the feature area is within a predetermined range, the adjacent area is also detected as the feature area.

[0030] The calculation unit 302 calculates exposure-related parameters for each of multiple regions on the imaging surface of the image sensor 201a based on the luminance of the first image captured by the imaging unit 201. An example of a specific calculation method will be described. First, the calculation unit 302 acquires a luminance histogram for the first region. Next, the calculation unit 302 calculates multiple exposure-related parameters to be applied to the first region so that the median value of the luminance histogram is gray (when the luminance gradation is an 8-bit histogram, the luminance is 128 or thereabouts).

[0031] The prediction unit 303 predicts a second region in a second image captured after the first image. The second region is a region corresponding to the first region. For example, if the first region detected by the detection unit 301 is a subject region, the prediction unit 303 predicts the subject region in the second image before the second image is captured by the imaging unit 201. The prediction unit 303 predicts the position of the second region. The prediction unit 303 predicts the position of the subject region (second region) in the second image corresponding to the subject region (first region) detected in the first image. The position of the second region can be predicted, for example, from multiple images captured by the imaging unit 201 at different times. That is, the movement direction and amount of movement of the subject region (first region) are calculated by calculating a movement vector from multiple images captured at different times. Then, it is possible to predict the position of the subject region (second region) in the second image captured by the imaging unit 201. The multiple images used to calculate the movement vector may or may not include the first image. It is preferable that the multiple images used to calculate the movement vector are captured after the first image. If the second image is an image captured immediately after (immediately after) the first image, a movement vector may be calculated from multiple images captured before the first image to predict the second region. Alternatively, the second region may be predicted using a method other than a movement vector. For example, a method may be used to predict the second region from the first image using a deep learning model. Alternatively, if the first region detected by the detection unit 301 is a region including a person or a vehicle, it is possible to predict the position of the second region (the region including the person or the vehicle) in the second image by extracting the orientation of the human body or face or the traveling direction of the vehicle.

[0032] The control unit 304 controls the imaging unit 201 to apply the exposure-related parameters to be applied to the first region calculated by the calculation unit 302 to the second region predicted by the prediction unit, and capture a second image.

[0033] The communication unit 305 performs overall control of communication between the imaging device 101 and the information processing device 103 via the network 102. For example, the communication unit 305 controls the timing at which images captured by the imaging device 101 are sent to the information processing device 103, receives operation information for the imaging device 101 sent from the information processing device 103, and controls the transmission of the information to each unit constituting the imaging device 101.

[0034] (Operation description) The operation of the imaging device according to this embodiment will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the operation of the imaging device according to this embodiment. The operation shown in this flowchart begins when a program stored inside or outside the imaging device 101 is loaded into the RAM 205 and the CPU 204 starts executing the loaded program.

[0035] In this embodiment, a case will be described in which two images including a first image are captured, the position of a second region in the second image is predicted, and the second image is captured as a third image. In this embodiment, the three consecutive frame images captured are respectively referred to as a t-2 image, a t-1 image (first image), and a t image (second image). Additionally, the t-2 image and the t-1 image are images captured at different times. In this embodiment, the t-2 image, the t-1 image, and the t image are three consecutive frames, but the images do not necessarily need to be consecutive.

[0036] In S501, the t-2 image is captured. At this time, the exposure parameters to be applied to each of the multiple regions on the imaging surface of the image sensor 201a are calculated based on the luminance of the image in the previous frame. Note that the exposure parameters to be applied may be calculated individually for each region, or exposure parameters to be applied uniformly to all regions may be calculated.

[0037] In S502, the detection unit 301 detects a characteristic region from the t-2 image captured by the imaging unit 201. Whether or not a characteristic region has been detected is determined in S502. The detected characteristic region is a region detected in the same manner as the first region and the second region. That is, the first region is a characteristic region in the first image (t-1 image), and the second region is a characteristic region in the second image (t image). In this embodiment, the characteristic region in the t-2 image is simply referred to as a characteristic region. If it is determined in S502 that a characteristic region has been detected, the process proceeds to S503; if not, the process returns to S502, and the t-2 image is captured again.

[0038] In S503, the calculation unit 302 calculates exposure-related parameters for each of multiple regions on the imaging surface of the image sensor 201a based on the luminance of the t-2 image. A map indicating the exposure-related parameters to be applied to each of multiple regions is called an exposure condition MAP. That is, in S503, the exposure condition MAP is created. In S503, the control unit 304 also applies the exposure condition MAP to the imaging surface of the image sensor 201a to control imaging by the imaging unit 201. The control of imaging by the control unit 304 includes control of setting exposure parameters for each of multiple regions on the imaging surface of the image sensor 201a, and control of the aperture, optical zoom, and focus of the imaging optical system 200 connected to the imaging device 101.

[0039] In S504, the image capturing unit 201 controlled by the control unit 304 captures the t-1 image.

[0040] In S505, the detection unit 301 detects a feature region (first region) from the t-1 image (first image). If a feature region is detected, the process proceeds to S506, and if a feature region is not detected, the process returns to S501.

[0041] In S506, it is determined whether the detected feature region is similar to the feature region detected in the t-2 image. The similarity determination is performed by comparing the image brightness of the feature region and the feature amount obtained by image analysis. If the degree of similarity is equal to or greater than a predetermined value, the feature region detected in the t-1 image is determined to be similar to the feature region detected in the t-2 image. Alternatively, similarity determination can be performed by comparing the positional relationship and size of the feature region in the t-2 image with the feature region in the t-1 image and determining whether they are adjacent or overlapping. If similarity is determined, the process proceeds to S507. If dissimilar, the process returns to S501. The detection unit 301 may also be configured to detect a region similar to the feature region detected in the t-2 image as the first region in the first image based on the feature amount or brightness of the feature region detected in the t-2 image. Therefore, the detection and determination operations may be performed separately or together. That is, S505 and S506 may be performed simultaneously or separately. In this embodiment, an example in which the determination is made individually will be described, but the determination may be made by the detection unit 301 or by a determination unit (not shown).

[0042] In S507, the prediction unit 303 predicts the position of the feature region (second region) in the t image (second image) captured after the t-1 image (first image). This prediction can be performed using various methods as described above. In this embodiment, as an example, a method of prediction using the motion vector of the feature region between the t-2 image and the t-1 image will be described. The motion vector can be calculated using various methods known in the art. For example, one method is to calculate the centroid position of the feature region in the t-2 image and the centroid position of the feature region in the t-1 image, and calculate a vector connecting these centroid positions as the motion vector. The centroid position of the feature region in the t image is predicted based on the direction and magnitude of the motion vector calculated using such a method. Furthermore, the size of the feature region in the t image is predicted based on the size of the feature region in the t-2 image or the t-1 image, the calculated motion vector, the position of the feature region in each image, etc. The prediction unit 303 may predict only the position of the second region, or may predict both the position and size separately.

[0043] In S508, the calculation unit 302 calculates exposure parameters based on the brightness of the t-1 image (first image) and creates an exposure condition MAP. In particular, the calculation unit 302 calculates exposure parameters for the first region based on the brightness of the first region in the first image.

[0044] In S509, the control unit 304 applies the exposure-related parameters calculated by the calculation unit 302 based on the luminance of the first region to the second region predicted by the prediction unit 303. In other words, the exposure condition MAP created in S508 is corrected so that the exposure parameters calculated based on the luminance of the first region in the exposure condition MAP are applied to the second region predicted by the prediction unit 303. Then, the control unit 304 controls the imaging unit 201 to capture a second image.

[0045] In S510, under the control of the control unit 304, the imaging unit 201 captures the t image (second image).

[0046] Creation of an exposure condition map for the t image (second image) in the imaging device according to this embodiment will be described with reference to FIG. 6. The designations t-2, t-1, and t in this figure correspond to those in FIG. 5. The shading of each region schematically indicates the magnitude of pixel values, and the shaded areas indicate feature regions detected by the detection unit 301 (and predicted by the prediction unit 303 for the t image). Furthermore, for convenience, symbols 1 through 5 and a through h are assigned to indicate the positions of multiple regions on the imaging plane 201a. For example, suppose the detection unit 301 detects a feature region in the t-2 image at position (c, 4) and a feature region in the t-1 image at position (d, 3). If the feature regions in the t-2 image and the t-1 image are determined to be similar, the prediction unit 303 predicts the feature region (second region) in the t image. The bottom of FIG. 6 shows the motion vector calculated by the prediction unit 303 and a predicted motion vector predicting the motion of the second region. In this way, a predicted motion vector can be calculated from the motion vectors of the feature region in the t-2 image and the t-1 image, and the position of the second region (feature region) in the t-2 image can be predicted. In this case, the position (e, 2) is predicted as the position of the second region. Based on the position of the second region predicted by the prediction unit 303 and the exposure condition map created by the calculation unit 302 in S508, the control unit 304 creates an exposure condition map based on the predicted position of the feature region and applies it to the imaging unit 201 for control. The exposure condition map based on the predicted position of the feature region is shown in the t image of FIG. 6. As shown in the exposure condition map when capturing the t image in FIG. 6, the exposure parameters calculated based on the luminance of the first region are applied to the predicted position of the second region, not to the same position as the first region. Note that the exposure conditions that were originally intended to be applied to the same position as the first region ((d, 3)) are instead the average of the exposure conditions of the adjacent surrounding regions, excluding the position of the second region. An exposure condition map may be created when the scale of the movement vector predicted by the prediction unit 303 relative to the first image or exposure area satisfies a predetermined condition. For example, if the magnitude of the movement vector is equal to or greater than a predetermined value based on the size of one exposure area, an exposure condition map is created, and if it is less than the predetermined value, an exposure condition map is not created.This makes it possible to avoid creating an exposure condition map when the amount of movement of the feature region between the first image and the second image is negligible.

[0047] According to the imaging device of this embodiment, when capturing an image of a moving object using an imaging element that can set exposure parameters for each of multiple regions on the imaging surface, it is possible to suppress deviations in exposure conditions for each region caused by movement of the image of the moving object between frames. In other words, even if the luminance distribution changes over time due to movement of the subject or light source, it is possible to capture images by setting appropriate exposure parameters for each of the multiple regions on the imaging surface.

[0048] In this embodiment, each functional block except for the imaging unit 201 may be included in the information processing device 103, in which case operations other than the imaging operation in this embodiment are executed by a CPU, RAM, and ROM (not shown) in the information processing device 103. At this time, an acquisition unit (not shown) in the information processing device 103 acquires the t-2 image and the t-1 image (first image) from the imaging device 101 via the network 102 and executes each operation.

[0049] <Variation 1> A modified example of this embodiment will be described with reference to Fig. 7. The imaging device according to this modified example has the same device configuration and functional configuration. In this modified example, a case will be described in which two images, a first image and a second image, are captured instead of three images.

[0050] (Operation description) The operation of the imaging device according to this modification will be described below. Note that S703, S705, and S706 correspond to S508, S509, and S510 in Fig. 5, respectively, and therefore descriptions thereof will be omitted.

[0051] In S701, a first image is captured. At this time, exposure-related parameters set for each of the multiple regions of the image sensor 201a may be controlled independently or uniformly.

[0052] In S702, the detection unit 301 detects the first region from the first image. If the first region is detected, the process proceeds to S703, and if not, the process returns to S701.

[0053] In S704, the prediction unit 303 predicts the second region in the second image from the first image. A specific example of predicting the position of the second region is prediction of movement of the subject region (characteristic region) using deep learning. In addition, the movement direction of the subject can be predicted by detecting the orientation of the subject in the characteristic region detected by the detection unit 301. For example, in the case of a person, a rough prediction of the movement distance can be made based on statistical average movement speeds of people or automobiles, and the position and size of the second region can be predicted based on the predicted movement direction and movement distance. In addition, the position and size of the moving object (second region) in the second image may be predicted using various other methods.

[0054] <Embodiment 2> In the first embodiment, the method for determining a characteristic region is modified as follows. FIG. 8 is a diagram illustrating a second embodiment for determining a characteristic region. For example, when capturing an image of a luminous subject, differences in the exposure conditions set for the subject and its surroundings occur. Particularly when the differences are significant, the range of subject illumination that can be captured between the exposure conditions of the exposure region corresponding to the subject and the exposure conditions of the surrounding exposure regions becomes discontinuous. In such cases, AE processing must be performed again each time the subject crosses an exposure region. Identifying this region as a characteristic region shortens the time required to determine the appropriate exposure conditions, enabling more optimal imaging. The detection unit 301 recognizes the exposure conditions for each exposure region as a range of luminance values ​​that can be captured, and determines whether a continuous luminance value range can be captured between a given exposure region and the exposure conditions set for its surroundings. Continuity is determined when there is a difference of more than one step. Specifically, the step difference in the exposure conditions is demodulated and corrected from the imaging results, and when the pixel values ​​of the entire image are consistent, the pixel values ​​of adjacent regions are considered continuous if either the upper or lower limit of the pixel value of the other region is within a range expanded by 1.

[0055] If the number of areas determined to be discontinuous among the eight exposure areas surrounding a certain area of ​​interest (exposure area of ​​interest) is equal to or greater than a fourth threshold, the area of ​​interest is determined to be a feature area.

[0056] Figure 9 shows a specific example of how adjacent regions can be determined to be contiguous. For example, if 8-bit imaging is possible for each region, the minimum shutter time and gain conditions are set for region A, and for region B adjacent to region A, a shutter time twice the minimum shutter time and minimum gain are set. In this case, to convert the image into an optimal one without any step differences between regions A and B, the pixel values ​​of region A must be doubled. That is, region B has a gradation range from 0 to 255, and region B has a gradation range from 256 to 511. In this case, because the upper limit value of region A and the lower limit value of region B are consecutive numbers, the two regions can be determined to be contiguous. Using the above method, the continuity of a region of interest with its surroundings is determined. If it is determined that there is no continuity with, for example, four or more adjacent regions, the region is determined to be a feature region.

[0057] According to this embodiment, even if a moving subject stops, it can continue to be determined as a feature region, and suitable imaging is possible even if the subject starts moving again.

[0058] <Other embodiments> The present invention can be realized by a process of reading and executing a program that realizes one or more functions of the above-described first embodiment. This program is supplied to a system or device via a network or a storage medium, and is read and executed by one or more processors in the computer of the system or device. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0059] 100 Imaging System 101 Imaging device 103 Information processing equipment 201 Imaging unit 301 Detector 302 Calculation Unit 303 Prediction Department 304 Control Unit

Claims

1. an imaging unit including an imaging element capable of setting exposure parameters for each of a plurality of regions on an imaging surface; a detection unit that detects a first region in a first image captured by the imaging unit; a calculation unit that calculates the exposure-related parameter based on the luminance of the first region in the first image; a prediction unit that predicts a second region in a second image captured after the first image, the second region being similar to the first region; a control unit that applies the exposure-related parameter calculated based on the luminance of the first region to the second region predicted by the prediction unit and controls the imaging unit to capture the second image; An imaging device comprising:

2. The imaging device according to claim 1 , wherein the detection unit detects an area in the first image that satisfies a predetermined condition as the first area.

3. The imaging device described in claim 1 or 2, characterized in that the detection unit detects, as the first region, a region in the first image corresponding to each of the plurality of regions, in which the number of pixels having a pixel value equal to or less than a first threshold or the number of pixels having a pixel value equal to or greater than a second threshold is equal to or greater than a third threshold.

4. 4. The imaging device according to claim 1, wherein the detection unit detects an area adjacent to the first area as the first area based on a comparison result between the exposure-related parameters of the area adjacent to the first area detected by the detection unit and the exposure-related parameters of the first area.

5. 5. The imaging device according to claim 1 , wherein the prediction unit predicts the second region in the second image based on a plurality of images captured at different times before the second image.

6. 6. The imaging device according to claim 5, wherein the prediction unit calculates a motion vector in the first region based on the plurality of images captured at different times, and predicts the second region based on the motion vector.

7. The imaging device according to claim 1 , wherein the prediction unit predicts the position of the second region.

8. The imaging device according to claim 7 , wherein the prediction unit predicts the size of the second region.

9. 9. The imaging device according to claim 1, further comprising an extracting unit that extracts a feature amount in the first region, and predicts the second region based on the feature amount.

10. The imaging device according to any one of claims 1 to 9, characterized in that the control unit applies the exposure parameters calculated based on the exposure parameters of an area adjacent to the first area to an area in the second image corresponding to the position of the first area.

11. 11. The imaging device according to claim 1, wherein the exposure-related parameters include at least one of an exposure time, an analog gain, and a digital gain.

12. An information processing device capable of communicating with an imaging device having an imaging element capable of setting exposure parameters for each of a plurality of regions on an imaging surface, an acquisition unit that acquires a first image captured by the imaging device; a detection unit that detects a first region in the first image; a calculation unit that calculates the exposure-related parameter based on the luminance of the first region in the first image; a prediction unit that predicts a second region in a second image captured after the first image, the second region being similar to the first region; a control unit that controls the imaging device to capture the second image by applying the exposure-related parameters calculated based on the luminance of the first region to the second region predicted by the prediction unit; and An information processing device comprising:

13. 1. A control method for controlling an imaging device having an imaging element capable of setting exposure parameters for each of a plurality of regions on an imaging surface, comprising: a detection step of detecting a first region in a first image captured by the imaging device; a calculation step of calculating the exposure-related parameter based on the luminance of the first region in the first image; a prediction step of predicting a second region in a second image captured after the first image, the second region being similar to the first region; a control step of applying the exposure-related parameters calculated based on the luminance of the first region to the second region predicted in the prediction step and controlling the imaging device to capture the second image; A control method comprising:

14. A program for causing a computer to execute the control method according to claim 13.

15. A computer-readable storage medium storing the program according to claim 14.

Images