Image capture device, image capture element, and method and program for controlling image capture device

Abstract

To perform an appropriate corrective process that suppresses the effect of high luminance light.SOLUTION: An imaging apparatus includes means for inferring the luminance of a high-luminance subject that is either equal to or more than the saturation level of a pixel signal from an image signal generated from a plurality of pixels and means for determining, on the basis of the inferred luminance of the high-luminance subject, an area for acquiring a correction value to be used in correcting the image signal.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging apparatus, an imaging element, and a control method and program for an imaging apparatus. [Background technology]

[0002] Image sensors such as CCD and CMOS mounted on imaging devices such as digital cameras are provided with optical black pixels (hereinafter, OB pixels) in a part of the pixel area. Black level correction processing for adjusting the black level to a reference value is performed using the output signal of the OB pixel. Here, when the black level correction processing using the output signal of the OB pixel is performed, high brightness light from a high brightness subject such as the sun may be incident on the vicinity of the OB pixel area. In this case, even though the OB pixel is shielded from light, light leaking from the adjacent aperture pixel may affect the OB pixel area. When the light leaking from the aperture pixel affects the OB pixel, the output signal of the OB pixel will no longer be a signal indicating a normal value, and the black level correction processing will not be performed properly. If the black level correction is not performed properly, unnatural coloring or the like may occur in the captured image.

[0003] Related techniques are proposed in Patent Documents 1 and 2. The technique in Patent Document 1 detects that the output level in the OB section changes according to the distance from the effective pixel section, and changes the OB region for calculating the dark level according to the detection result. The technique in Patent Document 2 performs gain correction on the AD conversion value of the pixel signal arranged in a column determined to have received light with a high luminance of a predetermined level or more, based on a gain value according to the distance from the position of the pixel where light with a high luminance of a predetermined level or more is received. In recent years, CNN (convolutional neural network) has been used for image recognition. A related technique is proposed in Patent Document 3. The technique in Patent Document 3 realizes feature extraction using CNN with a single sensor with a stacked structure. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2007-158830 A [Patent Document 2] JP 2014-175778 A [Patent Document 3] International Publication No. 2017 / 168665 Summary of the Invention [Problem to be solved by the invention]

[0005] For example, in an image sensor, shading may occur in the amount of dark current in a pixel region due to manufacturing variations or the like. In such a case, the technology of Patent Document 1 may erroneously detect light leakage to OB pixels. In addition, the technology of Patent Document 2 only detects high luminance light based on whether the output of a pixel signal is equal to or higher than a saturation level. The higher the luminance level, the more likely light leakage to OB pixels occurs, but even in a high-luminance subject with luminance equal to or higher than the saturation level, the pixel signal can only indicate up to the saturation level. Therefore, since the original luminance level of a high-luminance subject with luminance equal to or higher than the saturation level cannot be detected, the magnitude of the influence of light leakage from the output of the pixel signal to the OB pixels cannot be correctly determined. In such a case, correction processing such as black level correction processing may not be performed properly. Furthermore, the technology of Patent Document 3 does not solve the problem of performing correction processing such as black level correction processing properly.

[0006] Therefore, an object of the present invention is to perform appropriate correction processing that suppresses the influence of high brightness light. [Means for solving the problem]

[0007] In order to achieve the above object, the imaging apparatus of the present invention includes a means for inferring the luminance of a high-luminance subject that is equal to or higher than a saturation level of a pixel signal from an image signal generated from a number of pixels, and a means for detecting the luminance of the high-luminance subject based on the inferred luminance of the high-luminance subject. ,before Among the areas corresponding to optical black pixels included in the plurality of pixels, an area for acquiring a correction value used when correcting a black level of the image signal based on an output signal of the optical black pixel;and determining a region to be excluded from the region for obtaining the correction value. Effect of the Invention

[0008] According to the present invention, it is possible to perform appropriate correction processing that suppresses the influence of high brightness light. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a configuration of an imaging device. [Diagram 2] FIG. 2 is a diagram illustrating an example of an imaging element according to the first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of a pixel region. [Figure 4] FIG. 4 illustrates an example of a black level correction unit. [Diagram 5] FIG. 13 is a diagram illustrating an example of high luminance detection processing using a learning model. [Figure 6] FIG. 4 illustrates an example of a correction parameter determination unit. [Figure 7] FIG. 4 is a diagram illustrating an example of a correction value acquisition region. [Figure 8] FIG. 11 is a diagram illustrating an example of a feedback gain setting. [Figure 9] 5 is a flowchart showing an example of the flow of imaging processing in the first embodiment. [Figure 10] FIG. 2 illustrates an example of a learning unit according to the first embodiment; [Figure 11] 13 is a flowchart showing an example of the flow of a learning process. [Figure 12] FIG. 11 is a diagram illustrating an example of an imaging element according to a second embodiment. [Figure 13] 10 is a flowchart showing an example of the flow of imaging processing in the second embodiment. [Figure 14] FIG. 13 is a diagram illustrating an example of an imaging element according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited to the configurations described in the embodiments.

[0011] First Embodiment FIG. 1 is a diagram showing an example of the configuration of an imaging device 10. The imaging device 10 includes an imaging element 100, a signal processing unit 101, a photographing lens 102, a lens driving unit 103, a controller 104, a RAM 105, a display unit 106, a recording unit 107, and an operation unit 108. The imaging element 100 captures an object imaged through the photographing lens 102 as incident light. The imaging element 100 has a plurality of pixels, each of which includes a photodiode as a photoelectric conversion unit, arranged in a matrix. Each pixel of the imaging element 100 generates a pixel signal according to the incident light and outputs the generated pixel signal. The pixel signals output by the plurality of pixels constitute an image signal.

[0012] The photographing lens 102 forms an optical image of a subject on the image sensor 100. The lens driver 103 performs zoom control, focus control, aperture control, etc. on the photographing lens 102. The signal processor 101 performs signal processing such as rearrangement on the image signals output by the image sensor 100. The controller 104 is realized by a CPU or the like, and performs various controls. The controller 104 (CPU) executes a predetermined program to realize control as the controller 104 in each embodiment. The RAM 105 temporarily stores various information, image signals, etc. The display unit 106 displays various information, image signals, etc.

[0013] The recording unit 107 is a recording device such as a semiconductor memory for recording or reading out an image signal, and is detachable from the image sensor 100. The operation unit 108 includes buttons, dials, etc., and allows the user to perform various operations. The display unit 106 and the operation unit 108 may be a touch panel display integrally configured.

[0014] 2 is a diagram showing an example of the image sensor 100 of the first embodiment. The image sensor 100 includes an image signal generation unit 200, a memory 203, a high luminance detection unit 204, a black level correction unit 205, a correction parameter determination unit 206, and a learning unit 207. The image signal generation unit 200, the high luminance detection unit 204, the black level correction unit 205, the correction parameter determination unit 206, and the learning unit 207 can be realized by respective corresponding circuits.

[0015] The image signal generating unit 200 includes a pixel region 201 and an AD conversion unit 202. In the pixel region 201, a plurality of pixels (unit pixels) including a photodiode are arranged in a matrix. Each pixel is driven and controlled by a predetermined pixel control circuit. Each pixel converts incident light into an electric signal and outputs a pixel signal. The AD conversion unit 202 converts an analog pixel signal output from the pixel region 201 into a digital signal. For example, the AD conversion unit 202 has a plurality of AD converters corresponding to each column of the pixel region 201, and converts the analog pixel signal from the pixel region 201 into a digital signal in parallel on a row-by-row basis. The AD-converted digital pixel signal is output to a memory 203.

[0016] The memory 203 temporarily stores the digital image signal output by the AD conversion unit 202. The memory 203 is realized by, for example, a RAM or the like. The memory 203 can store image signals of a plurality of frames. The high brightness detection unit 204 detects the position and brightness of a subject having a predetermined brightness or more (high brightness subject) from the image signal temporarily stored in the memory 203. The high brightness subject is, for example, the sun. The predetermined brightness can be set to any brightness. For example, the predetermined brightness may be set to a brightness in a saturated state. In each embodiment, the high brightness detection unit 204 detects the position and brightness of a high brightness subject using a learning model (trained model) that has been machine-learned. The high brightness detection unit 204 also receives shooting condition information such as exposure time and aperture value of the shooting lens from the controller 104.

[0017] The black level correction unit 205 acquires the image signal temporarily stored in the memory 203, and performs correction (black level correction) based on the image signal of the OB pixel region so that the black level becomes a predetermined reference value. The black level corrected image signals are sequentially output to the outside of the image sensor 100. For example, the black level corrected image signal may be displayed on the display unit 106 or may be recorded in the recording unit 107. The correction parameter determination unit 206 determines the correction parameters of the black level correction unit 205 according to the detection result of the high brightness detection unit 204. The correction parameters include an area for acquiring a correction value (correction value acquisition area), a feedback gain, etc. The learning unit 207 performs machine learning of a learning model that detects the position and brightness information of a high brightness object from the image signal temporarily stored in the memory 203.

[0018] 3 is a diagram showing an example of the pixel region 201. The pixel region 201 includes an aperture pixel region 301 and an OB pixel region 302 (a region of optical black pixels). The OB pixel region 302 is provided on the outer periphery of the aperture pixel region 301. The aperture pixel region 301 is a region in which aperture pixels that receive an object image are arranged. The pixels included in the OB pixel region 302 have photodiodes optically shielded from light by an aluminum film or the like. The pixel signals of the OB pixel region 302 are used when acquiring a correction value for the black level correction unit 205.

[0019] 4 is a diagram showing an example of the black level correction unit 205. The black level correction unit 205 includes a subtraction unit 401, an OB signal acquisition unit 402, an error amount calculation unit 403, a feedback gain multiplication unit 404, a correction value calculation unit 405, a correction value acquisition area setting unit 406, and a feedback gain setting unit 407. The black level correction unit 205 performs black level correction on the image signal. The black level correction may be dark level correction. The image signals (input image signals) temporarily stored in the memory 203 are sequentially input to the black level correction unit 205.

[0020] The subtraction unit 401 subtracts the black level correction value calculated by the correction value calculation unit 405 from the pixel signal of the image signal input from the black level correction unit 205. The OB signal acquisition unit 402 acquires pixel signals of the OB pixel region 302 from the image signal subjected to the subtraction process by the subtraction unit 401, and outputs the acquired pixel signals to the error amount calculation unit 403. The OB signal acquisition unit 402 acquires a pixel signal of an OB pixel set by the correction value acquisition region setting unit 406 at a timing when the pixel signal of the OB pixel is input from among the pixel signals of the OB pixel region 302 subjected to the process by the subtraction unit 401. The region for acquiring the OB pixel set by the correction value acquisition region setting unit 406 is determined by the correction parameter determination unit 206. The OB signal acquisition unit 402 may output an average value of a plurality of OB pixels to the error amount calculation unit 403.

[0021] The error amount calculation unit 403 calculates the difference between a preset target value of the black level (black level reference value) and the current output level of the OB pixel output by the OB signal acquisition unit 402 as the error amount. The feedback gain multiplication unit 404 multiplies the error amount calculated by the error amount calculation unit 403 by a feedback gain (a tracking time constant of the black level correction process). It is desirable for the feedback gain multiplication unit 404 to set the feedback gain to a value smaller than "1" so that the black level correction is not excessive. The feedback gain setting unit 407 selects one from a plurality of preset feedback gains and outputs the selected feedback gain to the feedback gain multiplication unit 404. The selection of the feedback gain is determined by the correction parameter determination unit 206.

[0022] The correction value calculation unit 405 calculates a black level correction value by averaging the calculation results sequentially output from the feedback gain multiplication unit 404. As described above, the subtraction unit 401 subtracts the black level correction value calculated by the correction value calculation unit 405 from the pixel signal of the image signal input to the black level correction unit 205. The subtraction unit 401 subtracts the black level correction value, thereby performing black level correction processing on the image signal. The image signal that has been black level corrected is output from the black level correction unit 205 as an output image signal. By repeatedly performing the above black level correction processing, the amount of error between the signal level of the OB pixel and the black level reference value is gradually reduced, and finally, the output of the OB pixel as a correction result and the black level reference value converge to match. Since a correction value based on the OB signal has not been generated for the pixel signal that is input to the subtraction unit 401 first, a subtraction processing is performed with a predetermined correction value.

[0023] FIG. 5 is a diagram showing an example of high luminance detection processing using a learning model. The high luminance detection unit 204 includes a learning model 501 and a determination unit 506. The learning model 501 may have the function of the determination unit 506. The learning model 501 in FIG. 5 shows an example of a model using a CNN (convolutional neural network). The learning model 501 adopts a three-layer structure including an input layer 502, an intermediate layer 503, and an output layer 504. In the example of FIG. 5, an example in which the intermediate layer 503 is a single layer is shown, but a multi-layer structure may be adopted for the intermediate layer 503.

[0024] The input layer 502, the intermediate layer 503, and the output layer 504 include a plurality of nodes 505 for detecting whether or not a high-brightness object is included in an input image signal and for detecting the brightness level of the high-brightness object. Each node in the input layer 502 is connected to all the nodes in the intermediate layer 503. Each node in the intermediate layer 503 is connected to all the nodes in the output layer 504. In this way, the learning model 501 employs a neural network structure in which the nodes 505 are connected.

[0025] As shown in the example of FIG. 5, an image signal 500 corresponding to the aperture pixel region 301 among the image signals temporarily stored in the memory 203 is input as an input image signal to a plurality of nodes 505A in the input layer 502. The image signal 500 is divided into predetermined regions. Each region of the image signal 500 is input sequentially to each node 505A. The input layer 502 further includes nodes 505B, 505C, and 505D. Three pieces of shooting condition information, exposure time information, shooting gain information, and lens information, are input to the nodes 505B, 505C, and 505D, respectively, from the controller 104. The shooting condition information related to the shooting conditions is not limited to the three pieces of information in FIG. 5.

[0026] The exposure time information is information that indicates the exposure time when the image signal temporarily stored in the memory 203 was captured. The shooting gain information is information that indicates the amplification ratio when the image signal is amplified by the image signal generating unit 200 when the image signal was captured. The lens information is information that indicates the aperture value, focal length, etc. of the shooting lens 102 when the image signal was captured.

[0027] The output from each of the nodes 505A to 505D of the input layer 502 is propagated through the intermediate layer 503 to the output layer 504. Each of the nodes in the input layer 502, intermediate layer 503, and output layer 504 is weighted so as to appropriately output the likelihood of whether a high-luminance subject is included and which luminance level it corresponds to, based on the area of ​​the image signal 500 and the shooting condition information. The weight of each node is adjusted by machine learning, which will be described later. Based on the input information, the weight of each node is adjusted so as to become higher as the area of ​​the input image signal 500 becomes more likely to be at the corresponding luminance level among the multiple luminance levels.

[0028] In the example of FIG. 5, the brightness level of a high-brightness subject that is equal to or higher than the saturation level is expressed in five stages. Brightness level A is the high-brightness subject with the highest brightness level, and the brightness levels of the high-brightness subject decrease in the order of brightness levels B to D. Brightness level E indicates that the subject is not a high-brightness subject. In the input image signal, pixel signals in areas where subjects corresponding to brightness levels A to D exist are all at the saturation level. In the example of FIG. 5, when one area of ​​the image signal 500 is input to the learning model 501, the result that there is a "95%" possibility that the brightness level of the high-brightness subject is "A" and a "5%" possibility that the brightness level is "B" is obtained.

[0029] The determination unit 506 determines whether the region of the input image signal 500 includes a high-luminance object, and if so, what is the luminance level, based on the numerical values ​​of the luminance levels A to E output from the output layer 504. As described above, the image signal input to the learning model 501 is divided into a plurality of regions. Then, the above-mentioned processing (inference processing) is performed by the learning model 501 for each region. The region of the image signal 500 input to the learning model 501 is specified. Furthermore, the inference processing by the learning model 501 infers whether the region includes a high-luminance object, and if so, what is the luminance level of the high-luminance object. Therefore, a detection result (high-luminance detection result information 508) indicating the position of the high-luminance object in the aperture pixel region 301 and the luminance level of the high-luminance object is obtained. In the example of FIG. 5, it is detected that one region of the image signal 500 corresponding to the aperture pixel region 301 is a high-luminance object with a luminance level A, and the other regions are not high-luminance objects. High luminance detection result information 508 is output to the correction parameter determination unit 206 .

[0030] The learning model 501 described above is machine-learned in advance to infer what luminance level a high-luminance subject has, for a high-luminance subject that causes saturation of an image signal. Details of the machine learning method will be described later. In the example of FIG. 5, an example is shown in which the luminance level of a high-luminance subject is inferred, but other indices may be used for the brightness of a high-luminance subject. In addition, inference by the learning model 501 may be performed not for the entire region of the image signal 500 corresponding to the aperture pixel region 301, but for example, limited to a region within a predetermined certain range from the OB pixel region 302.

[0031] The high luminance detection unit 204 may infer the position and luminance of a high luminance subject from the image signal 500 corresponding to the aperture pixel region 301, using a learning model that has been machine-learned using any machine learning algorithm other than CNN. For example, the high luminance detection unit 204 may infer the position and luminance of a high luminance subject, using a learning model that has been machine-learned using any machine learning algorithm such as a support vector machine or logistic regression.

[0032] The high luminance detection unit 204 may detect the position and luminance level of a high luminance subject using a method other than the learning model. For example, the high luminance detection unit 204 may detect the position and luminance level of a high luminance subject based on the luminance value of each pixel of the image signal. The high luminance detection unit 204 may specify an area including a pixel signal of a saturation level in the input image signal as the position of the high luminance subject, and may use only the luminance level of the high luminance subject in the specified area as the target of inference using the learning model 501.

[0033] 6 is a diagram showing an example of the correction parameter determination unit 206. The correction parameter determination unit 206 includes a correction value acquisition region determination unit 601 and a feedback gain determination unit 602. The correction value acquisition region determination unit 601 determines a region from which a correction value is to be acquired in the OB pixel region 302 by the black level correction unit 205, based on high luminance detection result information acquired from the high luminance detection unit 204. The correction value acquisition region determination unit 601 outputs the determined correction value acquisition region to the correction value acquisition region setting unit 406 of the black level correction unit 205 as correction value acquisition region information.

[0034] FIG. 7 is a diagram showing an example of a correction value acquisition region. In the aperture pixel region 301, the luminance level of each region is indicated as high luminance detection result information. The OB pixel region 302 is divided into a plurality of regions. The use region 701 is a region used for acquiring a correction value, and the non-use region 702 is a region not used for acquiring a correction value. For example, assume that a high luminance subject is detected in region 703 among the regions of the image signal corresponding to the aperture pixel region 301. In this case, the OB pixel region (region within a predetermined range from the high luminance subject) near region 703 is set as a region excluded from the region for acquiring a correction value.

[0035] Here, the OB pixel area excluded from the correction value acquisition area is set to be wider as the luminance level of the high-luminance object detected in each area of ​​the image signal corresponding to the aperture pixel area 301 increases. This enhances the effect of reducing the influence of light leakage. On the other hand, when the luminance level of the high-luminance object is low, a wide range of OB pixels can be used for black level correction. This makes it possible to suppress the influence of noise and the like contained in the image signal and perform black level correction with good accuracy.

[0036] 6 determines the amount of feedback gain used in the feedback gain multiplication unit 404 of the black level correction unit 205, based on the high luminance detection result information acquired from the high luminance detection unit 204. The determined amount of feedback gain is output to the feedback gain setting unit 407 of the black level correction unit 205 as feedback gain information.

[0037] FIG. 8 is a diagram showing an example of feedback gain settings. In the example of FIG. 8, the feedback gain is changed row by row. In the OB pixel region 302, a region 801 draws a large amount of dark current initially in black level correction, so the feedback gain is set to "1 / 2". In the region 802, the feedback gain is set to "1 / 8". This allows the black level to be adjusted by gently following the change in dark current in the pixel region. In the region 803, which is in the vicinity of a region 804 of the aperture pixel region 301 where a high-brightness object is detected, the feedback gain is set to "1 / 16".

[0038] The region 803 is also a region in which the number of OB pixels used by the correction value acquisition region determination unit 601 to acquire the correction value is set to be small. When the parameter number of OB pixels used to acquire the correction value is small, an error occurs in the black level correction due to the influence of noise contained in the image signal, etc., and horizontal stripe-like noise may occur in the region of the aperture pixel region 301. Therefore, the feedback gain determination unit 602 sets the feedback gain of the region 803 low, thereby reducing the occurrence of horizontal stripe-like noise. In other words, the feedback gain is set low in a region in which the number of OB pixels used to acquire the correction value is small. As described above, the correction parameters of the black level correction unit 205 are changed according to the high brightness detection result of the high brightness detection unit 204. This makes it possible to suppress the influence of high brightness light and appropriately correct the black level.

[0039] FIG. 9 is a flowchart showing an example of the flow of the photographing process of the first embodiment. Each process of the flowchart of FIG. 9 is started, for example, after a photographing instruction is issued. When a control signal is output from the controller 104 to the image sensor 100 in S901, exposure is started in the image sensor 100. When exposure is started, signals are accumulated in each pixel of the pixel region 201 of the image sensor 100. After exposure is completed, the image signal is read out to the AD conversion unit 202 in S902. At this time, the signal of each pixel of the pixel region 201 is sequentially read out to the AD conversion unit 202, which converts the signal of each pixel into a digital pixel signal. The converted digital pixel signal is temporarily stored in the memory 203 as an image signal.

[0040] In S903, the high luminance detection unit 204 acquires the image signal temporarily stored in the memory 203, and detects a high luminance object from the acquired image signal. This acquires high luminance detection result information indicating the position and luminance level of the high luminance object. The high luminance detection result information is output to the correction parameter determination unit 206. The correction parameter determination unit 206 determines the correction value acquisition area and the correction parameters of the feedback gain based on the acquired high luminance detection result information. The correction parameter determination unit 206 outputs information on the determined correction value acquisition area and feedback gain to the black level correction unit 205.

[0041] In S905, the black level correction unit 205 performs black level correction processing on the image signal temporarily stored in the memory 203, based on the correction value acquisition region information and feedback gain information acquired from the correction parameter determination unit 206. In S906, the black level correction unit 205 outputs the image signal that has been subjected to the black level correction processing to the controller 104 via the signal processing unit 101. The controller 104 outputs the acquired image signal to the display unit 106, the recording unit 107, etc. The display unit 106 displays the image signal received from the controller 104 as a captured image. In addition, the recording unit 107 records the image signal received from the controller 104 in a semiconductor memory, etc.

[0042] Next, machine learning of the learning model will be described. FIG. 10 is a diagram showing an example of the learning unit 207 of the first embodiment. The learning unit 207 includes a teacher data generating unit 1001 and a learning model adjusting unit 1002. At least two image signals 1003 and 1004 including a high-luminance object are input from the memory 203 to the learning unit 207 as learning data. The image signal 1003 and the image signal 1004 are image signals obtained by capturing the same scene, but the exposure conditions at the time of capturing are different. The image signal 1003 (first image signal) is an image signal captured under a proper exposure condition. The image signal 1004 (second image signal) is an image signal captured under an underexposure condition. The image signal 1004 is an image signal obtained by capturing the same scene as the image signal 1003, and is a darker image than the image signal 1003. In other words, the image signal 1004 is an image obtained by darkening the image signal 1003. Furthermore, the learning unit 207 receives, from the controller 104, information on the shooting conditions of the image signals 1003 and 1004 (lens information, shooting gain information, exposure time information).

[0043] Image signal 1003 and image signal 1004 including a high-brightness object are input from memory 203 to teacher data generation unit 1001. In addition, shooting condition information (lens information, shooting gain information, exposure time information) indicating the shooting conditions when image signal 1003 and image signal 1004 were captured is input from controller 104 to teacher data generation unit 1001. Teacher data generation unit 1001 detects the position of the high-brightness object from at least one of image signals 1003 and 1004 input from memory 203.

[0044] Also, the approximate luminance level of the high-luminance subject can be calculated based on the output level from the position of the high-luminance subject in image signal 1004 captured under underexposure conditions and the above-mentioned shooting condition information. Teacher data generating unit 1001 generates the position and luminance level of the high-luminance subject as teacher data (correct answer data) based on the comparison result between image signal 1003 and image signal 1004 and the shooting condition information of image signal 1003 and image signal 1004.

[0045] The learning model adjustment unit 1002 adjusts the weight of each node of the CNN (learning model 501). An image signal 1003 is input to the learning model adjustment unit 1002 as learning data. In addition, shooting condition information (lens information, shooting gain information, exposure time information) when the image signal 1003 was captured as learning data is input to the learning model adjustment unit 1002. The learning model adjustment unit 1002 adjusts the weight of each node of the CNN so as to reduce the error between the output when the image signal 1003 and its shooting condition information are input and the teacher data (position and brightness level of a high-brightness subject).

[0046] As a result, when an unknown image signal is input, the CNN as the learning model 501 can infer the position and brightness level of a high-brightness object in the unknown image signal with high accuracy. In addition, the image signal 1003 used by the learning model adjustment unit 1002 for machine learning also includes cases where the brightness level of the high-brightness object is saturated. As a result, even if an unknown image signal in which the high-brightness object is saturated is input to the CNN, the CNN can infer the position and brightness level of the high-brightness object with high accuracy. As described above, the position of the high-brightness object can be detected without using a learning model, so the CNN may be a learning model that infers only the brightness level of the high-brightness object.

[0047] FIG. 11 is a flowchart showing an example of the flow of the learning process. Each process in the flowchart of FIG. 11 is started based on, for example, an instruction from a user. The controller 104 may display, on the display unit 106, information indicating that the learning process is being performed and a message encouraging shooting with the exposure set to the correct exposure. In S1101, the controller 104 performs control to perform shooting with the correct exposure condition based on a shooting instruction from the user. When the user sets the setting of the imaging device 10 to the correct exposure using the operation unit 108 and instructs shooting, shooting with the correct exposure condition is performed. The controller 104 temporarily stores the image signal generated by the image signal generating unit 200 in the memory 203.

[0048] In S1102, the high luminance detection unit 204 determines whether or not the image signal temporarily stored in the memory 203 includes a saturated pixel signal. If the high luminance detection unit 204 determines No in S1102, the process returns to S1101. In this case, the image signal captured in S1101 is not used for machine learning performed by the learning unit 207. For this reason, the controller 104 may, for example, cause the display unit 106 to display a screen encouraging the user to capture a scene including a high-luminance subject.

[0049] If the high brightness detection unit 204 determines Yes in S1102, it advances the process to S1103. In S1103, the controller 104 performs control to perform shooting of the same scene under underexposure conditions based on a shooting instruction from the user. At this time, the controller 104 may display a screen on the display unit 106 that prompts the user to change shooting conditions such as shooting gain, exposure time, and aperture value of the shooting lens. In this case, the user uses the operation unit 108 to change the exposure conditions so that the exposure conditions become underexposure conditions and instructs shooting. As a result, shooting is performed under underexposure conditions. The controller 104 temporarily stores the image signal generated by the image signal generation unit 200 in the memory 203.

[0050] In S1104, the learning unit 207 performs machine learning using the image signals (the first image signal captured in S1101 and the second image signal captured in S1104) temporarily stored in the memory 203. When the learning model 501 of the learning unit 207 is a CNN, the weight of each node of the CNN is adjusted by machine learning. As a machine learning method, an error backpropagation method or the like may be applied. When the composition of the first image signal captured in S1101 differs from the composition of the second image signal captured in S1104, the machine learning is not performed appropriately. Therefore, when the composition of the first image signal differs from the composition of the second image signal, the learning unit 207 may be controlled not to perform machine learning.

[0051] In S1105, the controller 104 determines whether to end the learning process. For example, if the user inputs an instruction to end the learning process using the operation unit 108, the controller 104 determines Yes in S1105. If the controller 104 determines Yes in S1105, it ends the process of the flowchart in FIG. 11. If the controller 104 determines No in S1105, it returns the process to S1101. In this case, a process of newly capturing an image signal to be used for machine learning is started.

[0052] The high brightness detection unit 204, the black level correction unit 205, and the learning unit 207 may be provided in, for example, the controller 104, instead of the image sensor 100. In this case, the controller 104 performs learning processing and performs black level correction. In the above example, the machine learning of the learning unit 207 is performed based on an image signal obtained by shooting using the image capture device 10, based on an instruction from a user. In this regard, for example, before the image capture device 10 is shipped, a learning model on which the above-mentioned machine learning has been performed may be set in advance in the learning unit 207. In this case, each process in the flowchart of FIG. 11 can be omitted.

[0053] Furthermore, if the imaging device 10 has a communication function, the learning model on which the above-mentioned learning process has been performed may be acquired from an external device. The external device performs machine learning in S1105 of Fig. 11 using an image signal captured under a proper exposure condition and an image signal captured under an underexposure condition. The imaging device 10 acquires the machine-learned learning model from the external device via the communication function, and sets the acquired learning model in the learning unit 207. In this case, each process in the flowchart of Fig. 11 can be omitted.

[0054] According to this embodiment, even if the high-brightness object included in the image signal is in a saturated state, the position and brightness level of the high-brightness object can be detected with high accuracy. This makes it possible to suppress the influence of high-brightness light and perform black level correction processing appropriately. This embodiment can also be applied to correction processing other than black level correction processing (correction processing based on an image signal including a high-brightness object). The same applies to the second and third embodiments described below.

[0055] <Second embodiment> The second embodiment is preferably applied to live view photography, video photography, and the like, in which continuous photography is performed and the generated image signals are displayed sequentially on the display unit 106. In the second embodiment, the high brightness detection unit 204 can select an image signal to be used for detecting a high brightness subject from among an image signal that has been subjected to black level correction and an image signal that has not been subjected to black level correction. The black level correction unit 205 determines correction parameters for black level correction of the next frame based on the high brightness subject detected by the high brightness detection unit 204. In the second embodiment, signal processing such as detection of a high brightness subject can be performed on the image signal after black level correction, so that the influence of a decrease in dynamic range due to dark current during signal processing is reduced.

[0056] Fig. 12 is a diagram showing an example of an image sensor 100 according to the second embodiment. The configuration other than the switching unit 1201 is the same as that in Fig. 2, and therefore description thereof will be omitted. An image signal that has been AD converted by the AD conversion unit 202 is input to the switching unit 1201. The switching unit 1201 switches the output destination of the input image signal to either the black level correction unit 205 or the memory 203. During normal shooting operation, the switching unit 1201 outputs the image signal to the black level correction unit 205. The switching unit 1201 is realized by a circuit.

[0057] On the other hand, when the learning unit 207 performs machine learning of the learning model, the switching unit 1201 outputs the image signal to the memory 203. At the stage when the machine learning of the learning model is performed, there is a high possibility that the accuracy of detecting the position and luminance level of a high-luminance object is insufficient. Therefore, an image signal that has been subjected to black level correction using a correction parameter based on the position and luminance level of a high-luminance object with insufficient detection accuracy is not suitable for machine learning of the learning model. When the machine learning of the learning model is performed, the switching unit 1201 outputs the image signal to the memory 203. This makes it possible to prevent an inappropriate image signal from being used for machine learning of the learning model.

[0058] As described above, the switching unit 1201 outputs the image signal to the black level correction unit 205 during normal shooting operation. As a result, the black level correction unit 205 performs black level correction on the image signal and outputs the image signal after black level correction to the memory 203. The memory 203 temporarily stores the image signal after black level correction. Then, the high brightness detection unit 204 reads out the image signal temporarily stored in the memory 203 and detects the position and brightness level of the high brightness object. Information on the position and brightness level of the detected high brightness object (high brightness detection result information) is output to the correction parameter determination unit 206. The correction parameter determination unit 206 determines correction parameters such as a correction value acquisition area and a feedback gain based on the high brightness detection result information. The black level correction unit 205 performs black level correction on the image signal of the next frame using the determined correction parameters.

[0059] Fig. 13 is a flowchart showing an example of the flow of the shooting process of the second embodiment. Each process of the flowchart of Fig. 13 is started, for example, after an instruction to shoot a moving image is given. When each process of the flowchart of Fig. 13 is started, the switching unit 1201 sets the output destination of the input image signal to the black level correction unit 205. On the other hand, when the learning process as shown in the flowchart of Fig. 11 is performed, the switching unit 1201 sets the output destination of the input image signal to the memory 203.

[0060] In S1301, when a control signal is output from the controller 104 to the image sensor 100, exposure is started in the image sensor 100. When exposure is started, signals are accumulated in each pixel of the pixel region 201 of the image sensor 100. After exposure is completed, image signals are read out in S1302. At this time, signals from each pixel in the pixel region 201 are sequentially read out to the AD conversion unit 202, which converts the signals from each pixel into digital pixel signals. The converted digital pixel signals are output to the black level correction unit 205 via the switching unit 1201.

[0061] In S1303, the black level correction unit 205 performs black level correction processing on the image signals sequentially output from the AD conversion unit 202. Here, the black level correction unit 205 performs black level correction processing on the image signal of the first frame using preset correction parameters. On the other hand, for the second frame and thereafter, the black level correction unit 205 performs black level correction processing with correction parameters determined based on the image signal of the previous frame. The image signal that has undergone the black level correction processing is temporarily stored in the memory 203 for detection of a high-luminance subject, and is also output to the display unit 106 or the like outside the image sensor 100 for video display.

[0062] In S1304, the controller 104 determines whether to end video shooting. Whether to end video shooting is determined based on whether the user has input an instruction to end shooting via the operation unit 108. If the controller 104 determines Yes in S1304, it ends the processing of the flowchart in Fig. 13. If the controller 104 determines No in S1304, it advances the processing to S1305.

[0063] In S1305, the high luminance detection unit 204 reads out the image signal temporarily stored in the memory 203, and detects the position and luminance level of a high luminance subject. In S1306, the correction parameter determination unit 206 determines correction parameters such as a correction value acquisition area and a feedback gain based on the high luminance detection result information detected by the high luminance detection unit 204. The determined correction parameters are used for black level correction processing of the next frame. After S1306, the process returns to S1301.

[0064] As described above, for example, in moving image shooting, live view shooting, etc., the black level correction unit 205 determines the correction parameters for black level correction of the next frame based on the high brightness subject detected by the high brightness detection unit 204. This makes it possible to obtain the same effects as in the first embodiment and to reduce the influence of a decrease in dynamic range due to dark current during signal processing.

[0065] <Third embodiment> Next, a third embodiment will be described. In the second embodiment, the high brightness detection result information detected by the high brightness detection unit 204 is reflected in the correction parameters when performing black level correction on the next frame. Therefore, if the position of the high brightness subject changes between frames, the accuracy of the black level correction process performed by the black level correction unit 205 may decrease. Therefore, in the third embodiment, the correction parameters for the next frame are determined taking into account the movement information of the high brightness subject.

[0066] FIG. 14 is a diagram showing an example of the image sensor 100 of the third embodiment. The configuration of FIG. 14 is obtained by adding a motion detection unit 1401 to the configuration of FIG. 12. Other configurations are the same as those of FIG. 12, and therefore description thereof will be omitted. Position information of a high-luminance object detected by the high-luminance detection unit 204 is input to the motion detection unit 1401. The motion detection unit 1401 compares the positions of the high-luminance object between frames, and predicts the position of the high-luminance object in the next frame from the change. The position information of the high-luminance object in the next frame predicted by the motion detection unit 1401 is output to the correction parameter determination unit 206. The correction parameter determination unit 206 determines correction parameters such as a correction value acquisition area and a feedback gain, taking into account the position information (predicted position information) of the high-luminance object in the next frame output by the motion detection unit 1401. As a result, even if a high-luminance object moves between frames, black level correction processing can be performed with good accuracy.

[0067] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to the above-mentioned embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. The present invention can also be realized by supplying a program that realizes one or more functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors of a computer in the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0068] 10. Imaging device 100 Image sensor 106 Display section 204 High brightness detection section 205 Black level correction section 206 Correction parameter determination unit 207 Learning Department 1201 Switching section

Claims

1. A means for estimating the luminance of a high-luminance subject that is equal to or exceeds a saturation level of a pixel signal from an image signal generated from a plurality of pixels; an imaging device comprising: a means for determining, based on the inferred luminance of a high-luminance subject, a region corresponding to an optical black pixel included in the plurality of pixels from which a correction value used when correcting the black level of the image signal based on an output signal of the optical black pixel is obtained; and a means for determining a region to be excluded from the region from which the correction value is obtained.

2. The imaging device of claim 1 , wherein the brightness of the high-brightness subject is detected by a learning model machine-learned using a first image signal and a second image signal that is an image of the same scene as the first image signal and is darker than the first image signal.

3. The imaging device according to claim 2 , wherein the learning model is machine-learned based on information on shooting conditions when the first image signal was captured and information on shooting conditions when the second image signal was captured.

4. 4. The imaging device of claim 3, wherein machine learning of the learning model is performed using training data including the luminance of a high-luminance subject detected based on a comparison between the first image signal and the second image signal, shooting condition information when the first image signal was captured, and shooting condition information when the second image signal was captured.

5. 5. The imaging device according to claim 2, wherein the first image signal is an image signal captured under a correct exposure condition, and the second image signal is an image signal captured under a dark exposure condition of the same scene as the first image signal.

6. The imaging device according to claim 5 , wherein when a composition of the first image signal differs from a composition of the second image signal, the first image signal and the second image signal are not used in machine learning of the learning model.

7. A display unit, a message prompting the user to take a photograph under the appropriate exposure conditions is displayed on the display unit; The imaging apparatus according to claim 5 , wherein after the first image signal is acquired, a message is displayed on the display unit to prompt the user to shoot under exposure conditions darker than the appropriate exposure conditions.

8. The imaging device according to claim 1 , further comprising: determining an area to be excluded from an area for obtaining the correction value based on the inferred luminance of the high-luminance subject and a position of the high-luminance subject.

9. The imaging device according to claim 1 , wherein the area excluded from the area for obtaining the correction value is set to be wider as the luminance of the high-luminance subject increases.

10. 10. The imaging device according to claim 1, wherein a process of inferring the brightness of the high-brightness subject is performed for an area within a predetermined range from an area corresponding to the optical black pixel including the image signal.

11. The imaging device according to claim 1 , further comprising: a correction unit for correcting the image signal based on the correction value.

12. The imaging device according to claim 11 , wherein an image signal of a next frame is corrected based on the correction value.

13. A means for estimating the luminance of a high-luminance subject that is equal to or exceeds a saturation level of a pixel signal from an image signal generated from a plurality of pixels; an imaging element comprising: a means for determining, based on the inferred luminance of a high-luminance subject, a region corresponding to an optical black pixel included in the plurality of pixels from which a correction value used when correcting the black level of the image signal based on an output signal of the optical black pixel is obtained; and a means for determining a region to be excluded from the region from which the correction value is obtained.

14. inferring a luminance of a high-luminance object that is equal to or exceeds a saturation level of a pixel signal from an image signal generated from a plurality of pixels; A method for controlling an imaging device, comprising: a step of determining, based on the inferred luminance of a high-luminance subject, a region corresponding to an optical black pixel included in the plurality of pixels from which a correction value used when correcting the black level of the image signal based on an output signal of the optical black pixel is obtained, and a region excluded from the region from which the correction value is obtained.

15. A program for causing a computer to execute each unit of the imaging apparatus according to any one of claims 1 to 12.

Images