Imaging device, method and program

Abstract

To restrain deterioration of an imaging element even when a strong amount of light is continuously detected in a specific region of the imaging element.SOLUTION: An imaging device includes an imaging element, a mechanism capable of changing the relative position between the imaging element and a subject image, deterioration degree estimation means for estimating the degree of deterioration over time for each region of the imaging element, brightness value acquisition means for acquiring, as a first brightness value, a brightness value of a screen corresponding to a first region which is the region where the degree exceeds a predetermined first threshold value, a control amount calculation unit for calculating a first control amount of the mechanism for assigning a subject having a brightness value equal to or less than the predetermined first brightness value to the first region based on a brightness value distribution over the entire screen corresponding to the imaging element when the first brightness value exceeds a predetermined first brightness value in the brightness value distribution, and system control means for controlling the mechanism based on the first control amount.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, a method, and a program. [Background technology]

[0002] Some image sensors deteriorate when exposed to light. For example, one type of image sensor is the Complementary Metal-Oxide-Semiconductor (CMOS) sensor, which uses color filters used in general cameras. When a CMOS sensor is exposed to strong light such as sunlight, the microlenses and color filters in the areas exposed to the strong light deteriorate due to the effects of ultraviolet rays and heat, causing fading in only certain areas.

[0003] For example, a single photon avalanche diode (SPAD) sensor is one type of image sensor. A SPAD sensor counts photons that enter a pixel and converts the photon into an electric charge as soon as it enters the pixel. A SPAD sensor can detect the large electric charge generated by the avalanche amplification caused by the electrons as a signal. Because of the mechanism of the SPAD sensor, no noise is introduced during signal readout, so the subject can be clearly photographed without being affected by noise even in dark places. Therefore, SPAD sensors are expected to be widely used as sensors for surveillance applications, etc. When a SPAD sensor is exposed to strong light such as sunlight, the microlenses and color filters in the area exposed to the strong light deteriorate due to the effects of ultraviolet rays and heat, and the pixels that cause avalanche amplification also deteriorate.

[0004] One of the causes of image quality degradation in SPAD sensors is the local change in dark count rate (DCR) depending on the amount of strong light that is locally applied. DCR is the average count value of photons detected in the absence of incident light. Local changes in DCR cause image quality degradation. In addition, in SPAD sensors, degradation of microlenses and color filters occurs just like in CMOS sensors.

[0005] Due to their intended use, pan-head cameras and surveillance cameras are often operated for long periods of time with a fixed shooting angle, and there is a possibility that a strong light source such as sunlight may be included in a specific position within the fixed shooting angle. In such cases, deterioration progresses only in a specific area on the sensor, resulting in partial degradation of image quality.

[0006] For example, Patent Document 1 discloses a technology that reads the brightness value of each pixel of an image sensor, and when the detected value is equal to or greater than a threshold value, adjusts the magnification of the optical zoom and the magnification of the digital zoom section to reduce degradation of the image sensor. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2012-095082 A Summary of the Invention [Problem to be solved by the invention]

[0008] However, the technology disclosed in Patent Document 1 detects the amount of light by the integrated value of the entire image sensor, so when strong light is locally applied, the luminance value of the part where the strong light is locally applied may not be detected. Therefore, the technology disclosed in Patent Document 1 cannot avoid local deterioration of the image sensor. In addition, the technology disclosed in Patent Document 1 does not take into account the degree of deterioration due to the accumulation of strong light, and detects only the situation at that time.

[0009] Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to suppress deterioration of the image sensor even when a strong amount of light is continuously detected in a specific area of ​​the image sensor. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the imaging device of the present invention is characterized in comprising an imaging element, a mechanism capable of changing the relative position between the imaging element and a subject image, a deterioration degree estimation means for estimating a degree of deterioration over time for each region of the imaging element, a brightness value acquisition means for acquiring a first brightness value as a first brightness value, a brightness value of a screen corresponding to a first region, which is the region where the degree exceeds a predetermined first threshold, a control amount calculation unit for calculating, when the first brightness value exceeds a predetermined first brightness value in a brightness value distribution over the entire screen corresponding to the imaging element, a first control amount of the mechanism for assigning a subject whose brightness is equal to or less than the predetermined first brightness value to the first region, based on the brightness value distribution, and a system control means for controlling the mechanism based on the first control amount. Effect of the Invention

[0011] According to the present invention, even if a strong amount of light is continuously detected in a specific area of ​​the image sensor, deterioration of the image sensor can be suppressed. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 illustrates an example of the configuration of an imaging device. [Diagram 2] FIG. 2 is a diagram illustrating an example of the configuration of an imaging element. [Diagram 3] FIG. 13 is a diagram illustrating a relative position change control parameter acquisition unit. [Figure 4] 4 is a flowchart showing a processing procedure according to the first embodiment. [Diagram 5] FIG. 13 is a diagram showing accumulated luminance values ​​for each region. [Figure 6] FIG. 4 is a diagram illustrating an example of luminance value estimation according to the first embodiment. [Figure 7]FIG. 4 is a diagram illustrating an example of luminance value estimation according to the first embodiment. [Figure 8] FIG. 11 is a diagram showing the relationship between the luminance value and the control amount. [Figure 9] 10 is a flowchart showing a processing procedure according to a second embodiment. [Figure 10] FIG. 11 is a diagram illustrating an example of luminance value estimation according to the second embodiment. [Figure 11] FIG. 11 is a diagram illustrating an example of luminance value estimation according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] An imaging device according to an embodiment includes an imaging element, a mechanism, a deterioration degree estimation means, a brightness value acquisition means, a control amount calculation unit, and a system control means. The mechanism is capable of changing a relative position between the imaging element and a subject image. The deterioration degree estimation means estimates a degree of deterioration over time for each region of the imaging element. The brightness value acquisition means acquires a brightness value of a screen corresponding to a first region, which is a region whose degree exceeds a predetermined first threshold, as a first brightness value. When the first brightness value exceeds a predetermined first brightness value in a brightness value distribution of the entire screen corresponding to the imaging element, the control amount calculation unit calculates a first control amount of a mechanism to which a subject having a brightness value equal to or less than the predetermined first brightness value is assigned to the first region based on the brightness value distribution. The system control means controls the mechanism based on the first control amount.

[0014] The luminance value acquisition means may acquire, as the second luminance value, a luminance value of a screen corresponding to a second region, which is a region in which the degree is equal to or less than a predetermined second threshold. In this case, when the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, the control amount calculation unit calculates, based on the luminance value distribution, a second control amount of a mechanism to which a subject having a luminance value equal to or less than the predetermined second luminance value is assigned to the second region. Then, the system control means controls the mechanism based on the second control amount.

[0015] The mechanism may also be a mechanism for controlling at least one of an optical zoom, a lens shift, a shift of an imaging element, an imaging direction of an imaging device, and a roll of the imaging device.

[0016] Furthermore, when there are a plurality of candidates for the first control amount, the control amount calculation means may determine the first control amount based on the magnitude of the first control amount and the amount of change in the luminance value of the first region.

[0017] Furthermore, when there are a plurality of candidates for the second control amount, the control amount calculation means may determine the second control amount based on the magnitude of the second control amount and the amount of change in the luminance value of the second region.

[0018] Furthermore, when there is no candidate for the first control amount, the control amount calculation means does not need to calculate the first control amount.

[0019] Furthermore, when there is no candidate for the second control amount, the control amount calculation unit does not need to calculate the second control amount.

[0020] The imaging device according to the embodiment may further include an image processing unit. The image processing unit is capable of executing at least one of the following image processing: changing the cropping range of the image generated by the imaging element, transforming the image, and rotating the image. The image processing unit executes image processing so that the difference between the range of the image before the mechanism is controlled by the system control unit and the range of the image after the mechanism is controlled by the system control unit falls within a predetermined range.

[0021] The control amount calculation means may limit the first control amount based on a correction amount that can be processed by image processing.

[0022] Moreover, the control amount calculation means limits the second control amount based on a correction amount that can be processed by image processing.

[0023] Furthermore, the image processing means cancels the image processing when the mechanism is controlled by the system control means.

[0024] The imaging device according to the embodiment may further include a notification unit that notifies the user that the range shown in the image has changed due to the mechanism being controlled by the system control unit, or that the image quality of the image has changed due to image processing.

[0025] The method and the program according to the embodiment include a degradation degree estimation step, a luminance value acquisition step, a control amount calculation step, and a system control step. In the degradation degree estimation step, the degree of degradation over time for each region of the image sensor is estimated. In the luminance value acquisition step, the luminance value of the screen corresponding to the first region, which is a region whose degree exceeds a predetermined first threshold, is acquired as a first luminance value. In the control amount calculation step, if the first luminance value exceeds a predetermined first luminance value in the luminance value distribution of the entire screen corresponding to the image sensor, the following processing is executed. That is, in such a case, in the control amount calculation step, a first control amount of a mechanism capable of changing the relative position between the image sensor and the subject image is calculated based on the luminance value distribution so that a subject having a luminance value equal to or less than the predetermined first luminance value is assigned to the first region. In the system control step, the mechanism is controlled based on the first control amount.

[0026] In the method and program according to the embodiment, a luminance value of a screen corresponding to a second region, which is a region in which the degree is equal to or less than a predetermined second threshold, is acquired as a second luminance value. In the method and program, when the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, a second control amount of a mechanism for allocating a subject having a luminance value equal to or less than the predetermined second luminance value to the second region is calculated based on the luminance value distribution.

[0027] Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[0028] <First embodiment> 1 is a block diagram showing an example of the configuration of an imaging device 100 according to the first embodiment. The imaging device 100 includes an imaging lens 101, an aperture 102, an ND filter 103, an imaging element 110, an image processing unit 112, a memory control unit 113, a system control unit 120, a non-volatile memory 121, a system memory 122, and a system timer 123. The imaging device 100 further includes a memory 130, a power control unit 140, a power supply unit 170, an I / F 180, and a pan / tilt control unit 190.

[0029] The imaging lens 101 is a lens group including a zoom lens, a focus lens, and a shift lens, and forms a subject image. The aperture 102 is an aperture used to adjust the amount of light. The ND filter 103 is a neutral density filter.

[0030] The image sensor 110 has an image sensor, performs photoelectric conversion, and outputs digital image data. Here, an example is described in which a single photon avalanche diode (SPAD) sensor is used as the image sensor. The image sensor 110 also has functions such as controlling accumulation by an electronic shutter, changing gain, and changing readout speed, and a mechanism for shifting the image sensor in a direction perpendicular to the optical axis.

[0031] The image processing unit 112 performs image processing on image data from the imaging element 110 or the memory control unit 113. The image processing includes, for example, predetermined pixel interpolation processing, resizing processing such as enlargement / reduction processing, image rotation and geometric deformation, image cropping, detection processing of luminance information, color information, characteristic objects, etc., color conversion processing, gamma correction processing, digital gain addition processing, etc. Image processing methods include image processing by a dedicated arithmetic circuit, image processing by a 3D-LUT processing circuit, etc. Furthermore, the image processing unit 112 performs predetermined arithmetic processing using image data from the imaging element 110, and transmits the arithmetic result to the system control unit 120.

[0032] Based on the transmitted calculation results, the system control unit 120 performs exposure control, distance measurement control, white balance control, etc. As a result, the system control unit 120 performs through-the-lens (TTL) type autofocus (AF) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, etc.

[0033] The output data from the imaging element 110 is written to the memory 130 via the image processing unit 112 and the memory control unit 113. Alternatively, the output data from the imaging element 110 is written to the memory 130 via the memory control unit 113. The memory 130 stores image data from the imaging element 110 or the image processing unit 112. The memory 130 may also be used to temporarily store an image that has been processed by the image processing unit 112 and to return the image to the image processing unit 112 again for application of another image processing. The memory 130 has a storage capacity sufficient to store moving images and audio for a predetermined period of time.

[0034] The non-volatile memory 121 is an electrically erasable and recordable memory, such as an EEPROM (Electrically Erasable Programmable Read-Only Memory). Constants, programs, etc. for the operation of the system control unit 120 are stored in the non-volatile memory 121. The programs referred to here are programs for executing various flowcharts described later.

[0035] The system control unit 120 controls the imaging device 100. The system control unit 120 executes each process of the first embodiment described below by executing a program recorded in the above-mentioned non-volatile memory 121. The system memory 122 uses a RAM (Random Access Memory), and constants and variables for the operation of the system control unit 120, programs read from the non-volatile memory 121, and the like are expanded.

[0036] The system timer 123 is a timing unit that measures the time used for various controls and the time of a built-in clock.

[0037] The power supply control unit 140 is composed of a battery detection circuit, a DC-DC converter, a switch circuit for switching between blocks to be energized, and detects whether a battery is attached, the type of battery, and the remaining battery power. Based on the result of this detection and instructions from the system control unit 120, the power supply control unit 140 controls the DC-DC converter to supply the required voltage for the required period to each unit including the external recording medium 150.

[0038] The power supply unit 170 is made up of a primary battery such as an alkaline battery or a lithium battery, a secondary battery such as a NiCd battery, a NiMH battery, or a Li-ion battery, an AC adapter, etc. The I / F 180 is an interface for connecting to an external recording medium 150 such as a memory card or a hard disk, and an external display device 160. The external recording medium 150 is a recording medium such as a memory card for recording captured images and exchanging data with the outside, and a semiconductor memory or the like is used.

[0039] The pan / tilt / roll control unit 190 adjusts and rotates the optical axis direction by moving the entire imaging device 100 in at least one of the pan direction, tilt direction, and roll direction in response to an instruction from the system control unit 120. Alternatively, the pan / tilt / roll control unit 190 adjusts and rotates the optical axis direction by moving the entire imaging unit in at least one of the pan direction, tilt direction, and roll direction in response to an instruction from the system control unit 120. The entire imaging unit includes the imaging lens 101, the aperture 102, the ND filter 103, and the image sensor 110.

[0040] FIG. 2 is a diagram showing an example of the configuration of the image sensor 110 in FIG. 1. Hereinafter, the configuration of the image sensor 110 and a method of reading out an image signal from the image sensor 110 will be described with reference to FIG. 2. As shown in FIG. 2, the image sensor 110 has a plurality of pixels 201, a vertical scanning section 208, a column memory 209, and an output section 210. Note that, for the sake of explanation, FIG. 2 shows two pixels 201 having the same structure arranged in the row direction and two pixels in the column direction, but in reality, more pixels 201 are arranged. Furthermore, the two pixels 201 arranged in the column direction are connected to a vertical signal line 207.

[0041] The image sensor 110 is a light receiving element using a SPAD. Each pixel 201 has an avalanche photodiode (hereinafter referred to as APD) 202, a quench resistor 203, an inverter 204, a counter 205, and a read switch 206. The pixel 201 is covered with color filters of multiple colors to obtain a color image. Light of different wavelength bands that has passed through the color filters is incident on each pixel 201. There are various types of color filters, and they may be of a primary color system or a complementary color system, and the type is not limited as long as a color image can be obtained. As an example, a case where R, G, and B color filters are arranged in a Bayer array will be described here. R, G, and B mean red, green, and blue, respectively.

[0042] The APD 202 is an APD operated in Geiger mode to detect single photons. The Geiger mode is an operation mode of an APD operated by applying a reverse bias voltage larger than the breakdown voltage. When a reverse bias voltage Vbias larger than the breakdown voltage Va is applied to the APD 202 to enter the Geiger mode, carriers generated by the incidence of a single photon cause avalanche multiplication, generating a large current. Here, the reverse bias voltage Vbias is, for example, -20V.

[0043] The quench resistor 203 is a resistive element for stopping avalanche multiplication of the APD 202. When a photon is incident on the APD 202 and a current is generated by avalanche multiplication, a voltage drop occurs in the quench resistor 203. When this voltage drop causes the voltage of the cathode of the APD 202 to swing from the reverse bias voltage Vbias to a breakdown voltage Va or lower, the avalanche multiplication stops. After that, when the cathode of the APD 202 is charged via the quench resistor 203, the voltage returns to the reverse bias voltage Vbias, which is the voltage before the fluctuation.

[0044] In this way, one voltage signal pulse is generated in response to the incidence of a single photon at the cathode of the APD 202. At this time, the width of the voltage signal pulse changes in response to a time constant determined by the resistance value of the quench resistor 203. The APD 202 and the quench resistor 203 constitute a sensor means.

[0045] The inverter 204 as a buffer means functions as a shaping section, shapes the waveform of the voltage signal pulse generated by the APD 202 as described above, and outputs the waveform-shaped signal pulse to its own output terminal.

[0046] The counter 205 receives as an input the signal pulse output by the inverter 204 and counts the rising edges of the signal pulse. The inverter 204 is used as a buffer means because the counter 205 is configured to count rising edges. When a buffer means that does not invert polarity is used, the counter 205 only needs to count the falling edges of the signal pulse output by the buffer.

[0047] As an example, the counter 205 is configured to be able to count the number of pulses for 16 bits. In the following description, the "count value" refers to the value counted by the counter 205. In addition, the counter 205 receives a signal pulse res output by the vertical scanning unit 208, and the reset operation of the count value and the start timing of the count operation are controlled.

[0048] The read switch 206 is turned on in response to a signal pulse sel output sequentially for each row by a vertical scanning unit 208. The count value held in the counter 205 is written to a column memory 209 via a vertical signal line 207.

[0049] As described above, the vertical scanning unit 208 sequentially selects rows of pixels arranged in a matrix and outputs a signal pulse sel to each row. The vertical scanning unit 208 also outputs a signal pulse res that controls the reset operation of the count value of the counter 205 and the start timing of the count operation.

[0050] The count values ​​of each pixel 201 in the row selected by the vertical scanning unit 208 in response to the signal pulse sel are written to the column memory 209 via the vertical signal line 207, and the column memory 209 holds the count values ​​for each column. The vertical scanning unit 208 also sequentially selects the count values ​​of each pixel 201 held in the column memory 209 for each column, thereby sequentially outputting the count values ​​of each pixel 201 to the output unit 210.

[0051] The output unit 210 performs digital processing such as gain processing and signal rearrangement on the count values ​​of each pixel 201 output from the column memory 209 , and outputs the processed image signal to the outside of the image sensor 110 .

[0052] As described above, the SPAD sensor counts photons that enter the pixel 201 and converts the photons into an electric charge as soon as they enter the pixel 201. The SPAD sensor can detect, as a signal, a large electric charge that is generated when the electrons that make up the electric charge undergo avalanche amplification. Because of the way the SPAD sensor works, no noise is introduced during signal readout, so that the subject can be clearly photographed without being affected by noise even in dark places. For this reason, the SPAD sensor is expected to be used widely as a sensor for surveillance purposes, etc.

[0053] The first embodiment has an object to suppress local degradation in image quality even when a strong light source is included in a specific position within a fixed shooting angle of view.

[0054] In the first embodiment, an operation will be described in which the image processing unit 112 estimates the degree of deterioration of the image sensor 110, and the system control unit 120 performs control to change the optical zoom of the imaging lens 101 or to shift the image sensor 110 based on the result.

[0055] FIG. 3 is a diagram showing an example of the configuration of a relative position change control parameter acquisition unit 300 between a subject and the image sensor 110. FIG. 4 is a flowchart showing processing executed by the image processing unit 112 shown in FIG. 1. The relative position change control parameter acquisition unit 300 is provided in the image processing unit 112 of FIG. 1, and has a deterioration degree estimation unit 304, a non-volatile memory 303, a brightness value acquisition unit 302, a control necessity determination unit 305, and a control amount calculation unit 306. The relative position change control parameter acquisition unit 300 generates an optical zoom control parameter of the imaging device 100 and a shift parameter of the image sensor 110. The system control unit 120 acquires these parameters. An input image 301 is an image based on an output signal of the image sensor 110.

[0056] The non-volatile memory 303 records the accumulated luminance value for each region of the image sensor 110, and updates the accumulated luminance value by accumulating the luminance value for each region newly acquired by the luminance value acquisition unit 302. Fig. 5 is a diagram showing an example of the accumulated luminance value for each region of the image sensor 110. When the angle of view of the image capturing device 100 is fixed and the image shown in Fig. 5(A) is continuously captured, the accumulated luminance value becomes as shown in Fig. 5(B). The accumulated luminance value is an index of the degree of deterioration.

[0057] For simplicity of explanation, an example is given here in which the image sensor 110 is divided into 8x6 regions; however, the image sensor 110 may be divided into smaller regions, and each pixel may have an index of the degree of degradation.

[0058] A larger number indicates that the image sensor 110 is more deteriorated, and that the possibility of image quality degradation occurring is high. The upper part of the image shown in Fig. 5(A) has a high score because the number of input photons to the image sensor 110 is always large due to the influence of the sky and sunlight. On the other hand, the lower part of the image shown in Fig. 5(B) has a low brightness level even during the day, and the number of input photons to the image sensor 110 is small, so the score is low.

[0059] In step S401, the luminance value acquisition unit 302 acquires sensor deterioration information. For example, the luminance value acquisition unit 302 acquires the cumulative luminance value of each area recorded in the non-volatile memory 303.

[0060] In step S402, the degradation degree estimation unit 304 determines whether or not there is local degradation. Specifically, when the cumulative luminance value is higher than other regions and is equal to or greater than a threshold value, the degradation degree estimation unit 304 considers the region to be degraded. For example, in the situation shown in FIG. 5B, the degradation degree estimation unit 304 considers the frame 501 to be a degraded region. When there is no degraded region, the degradation degree estimation unit 304 ends the process. On the other hand, when there is a degraded region, the degradation degree estimation unit 304 advances the process to step S403.

[0061] In step S403, the brightness value acquisition unit 302 acquires the brightness value of each region.

[0062] In step S404, the control necessity determination unit 305 determines whether the luminance value of the deteriorated region in the distribution of the luminance values ​​of each region exceeds a predetermined first luminance value. If the control necessity determination unit 305 determines that the luminance value of the deteriorated region in the distribution of the luminance values ​​of each region exceeds the predetermined first luminance value, the process proceeds to step S405. On the other hand, if the control necessity determination unit 305 determines that the luminance value of the deteriorated region in the distribution of the luminance values ​​of each region is equal to or less than the predetermined first luminance value, the process ends.

[0063] In step S405, the control amount calculation unit 306 provisionally determines a control amount. This control amount is, for example, at least one of zoom control by the imaging lens 101, shift control of the image sensor 110, pan control by the pan / tilt / roll control unit 190, tilt control, and roll control. Then, the control amount calculation unit 306 estimates the luminance value of an area where the image sensor 110 is deteriorated when the angle of view is changed.

[0064] For example, an object captured at 80% of the height of the image sensor 110 at a zoom magnification of 2x will be captured at 40% of the height by changing the zoom magnification to 1x. Also, for example, an object captured at the center of the image sensor 110 will be captured at a position shifted 1cm from the center by shifting the image sensor 110 by 1cm. Also, for example, an object captured at the edge of the image sensor 110 will be captured at the center of the image sensor 110 by performing pan control by half the angle of view so that the object moves toward the center. The control amount calculation unit 306 estimates the luminance value when the object captured in the deteriorated area of ​​the image sensor 110 changes when zoom control, shift control, pan control, tilt control, and roll control are performed as described above.

[0065] 6 and 7 are diagrams showing an example of luminance value estimation in the first embodiment. FIG. 6(A) shows an example of an image. FIG. 6(B) shows the distribution of luminance values ​​of the image shown in FIG. 6(A). If the frame 601 is a frame with a large degree of degradation, a high-luminance object is captured in the frame 601 with a large degree of degradation, and the degradation of the frame 601 area will progress further. Therefore, if the zoom factor is increased by the imaging lens 101 to control the angle of view to the telephoto side to the angle of view shown in FIG. 7(A), the distribution of luminance values ​​will be as shown in FIG. 7(B). As a result, the luminance of the object captured in the frame 601 will be reduced, making it possible to suppress the progression of degradation of the frame 601 area.

[0066] In step S405, the control amount calculation unit 306 may estimate the luminance value by changing the control amount multiple times and search for the control amount that minimizes the luminance value, or may compare the luminance value and the control amount and determine the control amount that minimizes both. Fig. 8 is a diagram showing the relationship between the luminance value and the control amount. When comparing the luminance value and the control amount, as shown in Fig. 8, for example, there is a method of plotting the relationship between the luminance value and the control amount and selecting the control amount that is closest to the origin as shown in plot 701.

[0067] In step S406, the control amount calculation unit 306 determines whether the luminance value of the deteriorated region of the image sensor 110 estimated in step S405 is equal to or less than a predetermined second threshold. Here, the predetermined second threshold is a luminance value that is relatively low with respect to the luminance distribution in the screen. If the control amount calculation unit 306 determines that the luminance value of the deteriorated region of the image sensor 110 estimated in step S405 is equal to or less than the predetermined second threshold, the control amount calculation unit 306 advances the process to step S407. On the other hand, if the control amount calculation unit 306 determines that the luminance value of the deteriorated region of the image sensor 110 estimated in step S405 exceeds the predetermined second threshold, the control amount calculation unit 306 ends the process. Note that, if the control amount calculation unit 306 determines that the luminance value of the deteriorated region of the image sensor 110 estimated in step S405 exceeds the predetermined second threshold, the control amount calculation unit 306 may return the process to step S405.

[0068] In step S407, the system control means 120 performs angle of view change control using the control amount provisionally determined in step S405. For example, the system control means 120 performs at least one of zoom control by the imaging lens 101, shift control of the imaging element 110, and pan control, tilt control, and roll control by the pan / tilt / roll control unit 190.

[0069] As described above, according to the first embodiment, it is possible to assign a low-luminance subject to a deteriorated area of ​​the image sensor 110, thereby making it possible to suppress local deterioration.

[0070] In the description of the first embodiment, control is performed so that a low-luminance subject is assigned to one deteriorated region of the image sensor 110, but control may be performed so that a low-luminance subject is assigned to multiple deteriorated regions. In this case, the control amount may be set so that all regions have a lower luminance than the threshold value, or the control amount may be set so that the average luminance value of the multiple deteriorated regions or the weighted average of the luminance values ​​taking into account the degree of deterioration is minimized.

[0071] Furthermore, in the first embodiment, control is performed so that a low-luminance subject is captured in an area of ​​the image sensor 110 with a large degree of degradation, but control may be performed so that a high-luminance subject is captured in an area with a small degree of degradation.

[0072] Furthermore, the angle of view changes when the process of the first embodiment is performed, so the user may be notified of the change in the angle of view by the external display device 160 or the like.

[0073] <Second embodiment> In the first embodiment, a method for suppressing local degradation of the image sensor 110 has been described by controlling the image sensor 100, the imaging lens 101, and the image sensor 110 so that a high-brightness subject is not captured in an area of ​​the image sensor 110 where the degree of degradation is large. However, controlling the image sensor 100, the imaging lens 101, the image sensor 110, etc. results in an angle of view different from the angle of view that is originally intended to be captured.

[0074] Therefore, in the second embodiment, a method for suppressing local deterioration of the image sensor 110 while minimizing the change in the angle of view will be described.

[0075] Fig. 9 is a flowchart for suppressing local deterioration of the image sensor 110 in the second embodiment. Note that steps S801 to S804 are the same as steps S401 to S404 in Fig. 4, and therefore description thereof will be omitted.

[0076] The relationship between the angle of view control and the luminance value in the second embodiment will be described with reference to the diagrams shown in Fig. 10 and Fig. 11. Fig. 10 and Fig. 11 are diagrams showing an example of luminance value estimation in the second embodiment. In the image shown in Fig. 10(A), the luminance values ​​of each region are the values ​​shown in Fig. 10(B). Here, it is assumed that the frame 901 is the most deteriorated region. In other words, the subject reflected in the most deteriorated region of the frame 901 has the highest luminance value, and the deterioration of the frame 901 will progress further.

[0077] Therefore, in step S805, the control amount calculation unit 306 provisionally determines at least one control amount of the zoom control by the imaging lens 101, the shift control of the image sensor 110, and the pan control, tilt control, and roll control by the pan / tilt / roll control unit 190. However, here, the control amount is limited on the premise that the original angle of view will be restored by the image processing unit 112. The processing performed by the image processing unit 112 is at least one of image cropping processing for cropping a specific area of ​​the image, image resizing processing for enlarging or reducing the image, image rotation processing for rotating the image, and image geometric transformation processing for geometrically transforming the image.

[0078] For example, in an image cropping process, if the original image can be cropped to a minimum size of 1 / 2 in both length and width, the imaging lens 101 can control the zoom magnification to 1 / 2 of the current size. Also, for example, if there are surplus pixels in the periphery of the image that will not be used in the final output image, it is possible to perform shift control of the imaging element 110 and pan control and tilt control by the pan / tilt control unit 190 for the number of surplus pixels.

[0079] Here, as an example, a process flow will be described in which the angle of view is changed to a wide angle by optical zoom control by the imaging lens 101, and then the original angle of view is restored by image cropping and image resizing by the image processing unit 112. For this reason, in step S805, the zoom control amount of the imaging lens 101 is provisionally determined.

[0080] 4, in step S806, the control amount calculation unit 306 determines whether or not the luminance value of the deteriorated region of the image sensor 110 is equal to or less than a predetermined second threshold. If the control amount calculation unit 306 determines that the luminance value of the deteriorated region of the image sensor 110 is equal to or less than the predetermined second threshold, the control amount calculation unit 306 advances the process to step S807. On the other hand, if the control amount calculation unit 306 determines that the luminance value of the deteriorated region of the image sensor 110 exceeds the predetermined second threshold, the control amount calculation unit 306 ends the process.

[0081] In step S807, similar to step S407 shown in Fig. 4, the system control means 120 performs angle of view change control using the control amount provisionally determined in step S405. For example, the system control means 120 performs at least one of zoom control, shift control of the image sensor 110, pan control by the pan / tilt / roll control unit 190, tilt control, and roll control. Here, the optical zoom is controlled to the wide-angle side, so an image as shown in Fig. 11(A) is obtained. The luminance values ​​of each region of this image are distributed as shown in Fig. 11(B), and since the luminance value of the region of the frame 901, which is the most deteriorated, is reduced, deterioration of the region of the frame 901 can be suppressed.

[0082] In step S808, the image processing unit 112 cuts out the range indicated by the cut-out range 902 and performs resizing to the same number of pixels as the original in order to return the angle of view changed by the zoom control to the original angle of view. As a result, it is possible to return the angle of view to the same angle as in FIG. 10(A).

[0083] As described above, according to the second embodiment, it is possible to assign a low-luminance subject to a deteriorated area of ​​the image sensor 110, thereby suppressing local deterioration and further minimizing changes in the angle of view.

[0084] In the description of the second embodiment, control is performed so that a low-luminance subject is assigned to one deteriorated region of the image sensor 110, but control may be performed so that a low-luminance subject is assigned to multiple deteriorated regions. In this case, the control amount may be set so that all regions have a lower luminance than the threshold value, or the control amount may be set so that the average luminance value of the multiple deteriorated regions or the weighted average of the luminance values ​​taking into account the degree of deterioration is minimized.

[0085] In the second embodiment, control is performed so that a low-luminance subject is captured in an area of ​​the image sensor 110 with a large degree of degradation, but control may be performed so that a high-luminance subject is captured in an area with a small degree of degradation.

[0086] Furthermore, when the processing of the second embodiment is performed, although the change in the angle of view is small, image quality may be affected depending on the processing performed by the image processing unit 112. Therefore, when this control is performed, the user may be notified by the external display device 160 or the like that the image quality has changed.

[0087] In addition, if the user performs an operation to change the angle of view after this control has been performed, the relationship between the deteriorated area of ​​the image sensor 110 and the subject brightness will change, so the processing in the image processing unit 112 performed in this control may be reset.

[0088] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0089] The present invention also includes a case where a software program for implementing the functions of the above-described embodiments is directly supplied from a recording medium to a system or device having a computer capable of executing the program, and the program is executed.The present invention also includes a case where a software program for implementing the functions of the above-described embodiments is supplied to a system or device having a computer capable of executing the program via wired / wireless communication, and the program is executed.

[0090] Therefore, the program code itself that is supplied to and installed in a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

[0091] The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as a magnetic tape, an optical / magneto-optical storage medium, or a non-volatile semiconductor memory.

[0092] As a method for supplying the program, a method is also considered in which the computer program constituting the present invention is stored in a server on a computer network, and a connected client computer downloads the computer program.

[0093] The present invention includes the following inventions that appropriately combine the above-mentioned features.

[0094] (Configuration 1) an image pickup device comprising: an image pickup element; a mechanism capable of changing a relative position between the image pickup element and a subject image; a deterioration degree estimation means for estimating a degree of deterioration over time for each region of the image pickup element; a brightness value acquisition means for acquiring a brightness value of a screen corresponding to a first region, the first region being the region where the degree exceeds a predetermined first threshold, as a first brightness value; a control amount calculation unit for calculating, when the first brightness value exceeds a predetermined first brightness value in a brightness value distribution over the entire screen corresponding to the image pickup element, a first control amount of the mechanism for assigning a subject whose brightness is equal to or less than the predetermined first brightness value to the first region, based on the brightness value distribution; and a system control means for controlling the mechanism based on the first control amount.

[0095] (Configuration 2) The imaging device described in configuration 1, characterized in that the brightness value acquisition means acquires a brightness value of a screen corresponding to a second area, which is the area where the degree is equal to or less than a predetermined second threshold, as a second brightness value, and when the second brightness value is equal to or less than a predetermined second brightness value in the brightness value distribution, the control amount calculation unit calculates a second control amount of the mechanism for assigning a subject whose brightness value is equal to or less than the predetermined second brightness value to the second area based on the brightness value distribution, and the system control means controls the mechanism based on the second control amount.

[0096] (Configuration 3) 3. The imaging device according to configuration 1 or 2, wherein the mechanism controls at least one of an optical zoom, a lens shift, a shift of the imaging element, an imaging direction of the imaging device, and a roll of the imaging device.

[0097] (Configuration 4) The imaging device described in any one of configurations 1 to 3, characterized in that, when there are multiple candidates for the first control amount, the control amount calculation means determines the first control amount based on a magnitude of the first control amount and an amount of change in the luminance value of the first region.

[0098] (Configuration 5) The imaging device described in configuration 2, characterized in that when there are multiple candidates for the second control amount, the control amount calculation means determines the second control amount based on a magnitude of the second control amount and an amount of change in luminance value of the second region.

[0099] (Configuration 6) 6. The imaging device according to any one of configurations 1 to 5, wherein the control amount calculation means does not calculate the first control amount when there is no candidate for the first control amount.

[0100] (Configuration 7) 3. The imaging device according to configuration 2, wherein the control amount calculation section does not calculate the second control amount when there is no candidate for the second control amount.

[0101] (Configuration 8) The imaging device according to any one of configurations 1 to 7, further comprising an image processing means that is capable of executing at least one of changing a cropping range of an image generated by the imaging element, transforming the image, and rotating the image as image processing, and that executes the image processing so that a difference between a range shown in the image before the mechanism is controlled by the system control means and a range shown in the image after the mechanism is controlled by the system control means falls within a predetermined range.

[0102] (Configuration 9) 9. The imaging apparatus according to configuration 8, wherein the control amount calculation means limits the first control amount based on a correction amount that can be processed by the image processing.

[0103] (Configuration 10) 9. The imaging apparatus according to configuration 8, wherein the control amount calculation means limits the second control amount based on a correction amount that can be processed by the image processing.

[0104] (Configuration 11) 9. The imaging device according to configuration 8, wherein the image processing means cancels the image processing when the mechanism is controlled by the system control means.

[0105] (Configuration 12) 12. The imaging device according to any one of configurations 1 to 11, further comprising a notification means for notifying a user that a range shown in the image has changed due to the mechanism being controlled by the system control means.

[0106] (Configuration 13) 9. The imaging device according to any one of configurations 1 to 8, further comprising a notification unit that notifies a user that the image quality of the image has changed due to the image processing.

[0107] (Method 1) a first luminance value corresponding to a first region where the degree of deterioration over time exceeds a predetermined first threshold; and if the first luminance value exceeds a predetermined first luminance value in a luminance value distribution over the entire screen corresponding to the image sensor, a first control amount of a mechanism capable of changing a relative position between the image sensor and a subject image is calculated based on the luminance value distribution so that a subject with a luminance value equal to or lower than the predetermined first luminance value is assigned to the first region.

[0108] (Method 2) A method as described in method 2, characterized in that a luminance value of a screen corresponding to a second region, which is the region where the degree is equal to or less than a predetermined second threshold, is obtained as a second luminance value, and when the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, a second control amount of the mechanism for assigning a subject whose luminance value is equal to or less than the predetermined second luminance value to the second region is calculated based on the luminance value distribution, and the mechanism is controlled based on the second control amount.

[0109] (Program 1) a program that causes a computer to execute a process of estimating a degree of deterioration over time for each region of an image sensor, execute a process of acquiring a luminance value of a screen corresponding to a first region where the degree exceeds a predetermined first threshold as a first luminance value, and when the first luminance value exceeds a predetermined first luminance value in a luminance value distribution over the entire screen corresponding to the image sensor, execute a process of calculating a first control amount of a mechanism that can change the relative position between the image sensor and a subject image based on the luminance value distribution so that a subject with a luminance value equal to or lower than the predetermined first luminance value is assigned to the first region, and execute a process of controlling the mechanism based on the first control amount.

[0110] (Program 2) The program described in program 1 is characterized in that it causes a computer to execute a process of acquiring a luminance value of a screen corresponding to a second region, which is the region where the degree is equal to or less than a predetermined second threshold, as a second luminance value, and if the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, execute a process of calculating a second control amount of the mechanism to assign a subject whose luminance value is equal to or less than the predetermined second luminance value to the second region based on the luminance value distribution, and execute a process of controlling the mechanism based on the second control amount.

[0111] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a recording medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit for implementing one or more of the functions, such as an ASIC (Application Specific Integrated Circuit).

[0112] The above describes preferred embodiments of the present invention. However, the present invention is not limited to the above-described embodiments. In other words, the present invention includes embodiments in which various modifications have been made based on the gist of the present invention, and these embodiments are not excluded from the scope of the present invention. [Explanation of symbols]

[0113] 112: Image processing unit 120: System control unit 160: External display device 302: Brightness value acquisition unit 304: Deterioration degree estimation unit 305: Control necessity determination unit 306: Control amount calculation unit

Claims

1. An imaging element; a mechanism capable of changing a relative position between the image sensor and a subject image; a deterioration degree estimation means for estimating a degree of deterioration over time for each region of the image sensor; a brightness value acquiring means for acquiring, as a first brightness value, a brightness value of a screen corresponding to a first region, which is the region in which the degree exceeds a predetermined first threshold; a control amount calculation unit that calculates, when the first luminance value exceeds a predetermined first luminance value in a luminance value distribution across an entire screen corresponding to the image sensor, a first control amount of the mechanism for allocating an object having a luminance value equal to or less than the predetermined first luminance value to the first region based on the luminance value distribution; a system control means for controlling the mechanism based on the first control amount; An imaging device comprising:

2. the luminance value acquisition means acquires, as a second luminance value, a luminance value of a screen corresponding to a second region, which is the region in which the degree is equal to or smaller than a predetermined second threshold value; when the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, the control amount calculation unit calculates a second control amount of the mechanism for allocating an object having a luminance value equal to or less than the predetermined second luminance value to the second region based on the luminance value distribution; The system control means controls the mechanism based on the second control amount.

2. The imaging device according to claim 1 .

3. the mechanism controls at least one of an optical zoom, a lens shift, a shift of the image sensor, an imaging direction of the image capture device, and a roll of the image capture device; 3. The imaging device according to claim 1 or 2.

4. When there are a plurality of candidates for the first control amount, the control amount calculation means determines the first control amount based on a magnitude of the first control amount and an amount of change in the luminance value of the first region.

3. The imaging device according to claim 1 or 2.

5. When there are a plurality of candidates for the second control amount, the control amount calculation means determines the second control amount based on a magnitude of the second control amount and an amount of change in the luminance value of the second region.

3. The imaging device according to claim 2.

6. The control amount calculation means does not calculate the first control amount when there is no candidate for the first control amount.

3. The imaging device according to claim 1 or 2.

7. The control amount calculation unit does not calculate the second control amount when there is no candidate for the second control amount.

3. The imaging device according to claim 2.

8. the image processing device is further provided with an image processing means that is capable of executing at least one of changing a cutout range of an image generated by the imaging element, transforming the image, and rotating the image as image processing, and that executes the image processing so that a difference between a range shown in the image before the mechanism is controlled by the system control means and a range shown in the image after the mechanism is controlled by the system control means falls within a predetermined range; 3. The imaging device according to claim 2.

9. the control amount calculation means limits the first control amount based on a correction amount that can be processed by the image processing.

9. The imaging device according to claim 8.

10. the control amount calculation means limits the second control amount based on a correction amount that can be processed by the image processing.

9. The imaging device according to claim 8.

11. the image processing means cancels the image processing when the mechanism is controlled by the system control means; 9. The imaging device according to claim 8.

12. a notification unit that notifies a user that a range shown in the image has changed due to the mechanism being controlled by the system control unit, 9. The imaging device according to claim 8.

13. a notification unit for notifying a user that the image quality of the image has been changed by the image processing, 9. The imaging device according to claim 8.

14. Estimate the degree of deterioration over time for each area of ​​the image sensor, A luminance value of a screen corresponding to a first region, which is the region in which the degree exceeds a predetermined first threshold, is acquired as a first luminance value; when the first luminance value exceeds a predetermined first luminance value in a luminance value distribution over an entire screen corresponding to the image sensor, a first control amount of a mechanism capable of changing a relative position between the image sensor and a subject image is calculated based on the luminance value distribution so that a subject having a luminance value equal to or less than the predetermined first luminance value is assigned to the first region; controlling the mechanism based on the first control amount; 23. A method comprising:

15. A luminance value of a screen corresponding to a second region, which is the region in which the degree is equal to or less than a predetermined second threshold, is acquired as a second luminance value; when the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, a second control amount of the mechanism for allocating an object having a luminance value equal to or less than the predetermined second luminance value to the second region is calculated based on the luminance value distribution; controlling the mechanism based on the second control amount; 15. The method of claim 14.

16. On the computer, Executing a process for estimating the degree of deterioration over time for each region of the image sensor; execute a process of acquiring, as a first luminance value, a luminance value of a screen corresponding to a first region, which is the region in which the degree exceeds a predetermined first threshold value; executes a process of calculating a first control amount of a mechanism capable of changing a relative position between the image sensor and a subject image based on the luminance value distribution when the first luminance value exceeds a predetermined first luminance value in a luminance value distribution over the entire screen corresponding to the image sensor, so that a subject having a luminance value equal to or less than the predetermined first luminance value is assigned to the first region; executing a process of controlling the mechanism based on the first control amount; A program characterized by:

17. On the computer, acquiring, as a second luminance value, a luminance value of a screen corresponding to a second region, which is the region in which the degree is equal to or less than a predetermined second threshold value; when the second luminance value is equal to or less than a predetermined second luminance value in the luminance value distribution, a process is executed to calculate a second control amount of the mechanism for allocating an object having a luminance value equal to or less than the predetermined second luminance value to the second region based on the luminance value distribution; executing a process of controlling the mechanism based on the second control amount; 17. The program according to claim 16 .

Images