Solid-state imaging device

Abstract

[Problem] To suppress screen flicker. [Solution] This solid-state imaging device comprises a processing circuit. The processing circuit executes a process of: dividing a plurality of images acquired in a rolling shutter format and successive in the time series of acquisition such that, with respect to a first direction in which the shutter timing is the same and a second direction that intersects the first direction and along which the shutter timing is scanned, pixels belonging to at least the same first direction form the same block; estimating a frame period of flicker caused by the light source from a luminance value in the block; and suppressing the flicker caused by the light source on the basis of at least the frame period.

Description

solid-state imaging device

[0001] The present disclosure relates to a solid-state imaging device.

[0002] Artificial light sources such as fluorescent lamps and LEDs (Light Emitting Diodes) flicker repeatedly at a regular interval. When capturing an image under an artificial light source using a rolling shutter image sensor, stripes called in-plane flicker appear in the captured image, significantly degrading imaging performance.

[0003] LED lighting, which has become increasingly popular in recent years, is characterized by its high-speed flickering, as the light source frequency is not fixed for each device. Existing flicker correction methods for fluorescent lamps, whose flicker cycle is determined by the power supply frequency, have been able to avoid flicker by detecting the peak of the light source's flicker and precisely adjusting the exposure timing. However, with LED lighting, whose light source frequency is unknown and which flickers at high speed, it is difficult to accurately detect the flickering timing, i.e., to perform flicker correction.

[0004] One method of flicker correction for LED lighting is to synchronize the exposure time with the light source frequency. With this method, the user must manually set and search for the exposure time at which flicker disappears. Research is also being conducted into functions that automatically change the exposure time to search for the exposure time at which flicker disappears, but with either method, it takes time to search for the exposure time at which flicker disappears, making it difficult to correct flicker in a short time.

[0005] Japanese Patent Application Publication No. 2019-134257

[0006] Therefore, one non-limiting problem to be solved by the embodiments of the present disclosure is to estimate or suppress in-plane flicker in an image captured in a rolling shutter format. The problem to be solved by the embodiments of the present disclosure can also be, as some further non-limiting examples, a problem corresponding to the effects described in the embodiments. In other words, a problem corresponding to at least one of the effects described in the description of the embodiments of the present disclosure can be considered to be a problem to be solved by the present disclosure.

[0007] According to one embodiment, a solid-state imaging device includes a processing circuit that divides a plurality of chronologically consecutive images acquired using a rolling shutter method into blocks in a first direction in which the shutter timings are the same and a second direction that intersects with the first direction and scans the shutter timings, so that pixels belonging to at least the same first direction are grouped into the same blocks, estimates a frame period of flicker caused by a light source from luminance values ​​in the blocks, and performs processing to suppress flicker caused by the light source based on at least the frame period.

[0008] The processing circuitry may estimate the frame period based on changes in luminance values ​​in the block.

[0009] The processing circuit may calculate an autocorrelation value of changes in luminance values ​​in the blocks acquired in time series, and may estimate the period at which the autocorrelation value is maximized as the frame period.

[0010] The processing circuit may calculate a correction gain value for each block in each frame so that the luminance value of the block is the maximum value of the luminance value of the block in the frame period, may correct the luminance value of each pixel in each block based on the correction gain value calculated for the block, and may combine the blocks with corrected luminance values ​​within the frame to obtain an image with suppressed flicker.

[0011] The processing circuit may calculate an exposure time based on the frame period.

[0012] The processing circuitry may calculate the exposure time as a product of the frame period and a natural number.

[0013] The processing circuitry may further calculate an exposure time based on the correction gain value.

[0014] The processing circuit may calculate an in-plane flicker period that exists across the blocks for an image in a frame from the correction gain value for each of the blocks, and may calculate the exposure time based on the frame period and the in-plane flicker period.

[0015] The processing circuit may estimate a light source frequency based on the frame period and the in-plane flicker period, and may calculate the exposure time as a natural number multiple of the blinking period of the light source calculated from the light source frequency.

[0016] According to one embodiment, the solid-state imaging device further includes an imaging section that acquires frame images in a rolling shutter format.

[0017] When the imaging unit is included, the processing circuit can control the exposure time of the imaging unit based on the calculated exposure time.

[0018] 1 is a block diagram schematically showing a solid-state imaging device according to an embodiment; FIG. 2 is a diagram schematically showing an example of in-plane flicker according to an embodiment; FIG. 3 is a diagram schematically showing an example of block division according to an embodiment; A flowchart showing an example of correction processing according to an embodiment; FIG. 4 is a diagram schematically showing an example of luminance values ​​of blocks between frames according to an embodiment; FIG. 5 is a diagram schematically showing calculation of correction gain values ​​according to an embodiment; FIG. 6 is a diagram schematically showing gain correction of a frame image according to an embodiment; A flowchart showing an example of correction processing according to an embodiment; A flowchart showing an example of a part of correction processing according to an embodiment; A diagram schematically showing correction gain values ​​of each block in a frame image according to an embodiment; A diagram schematically showing correction of exposure time according to an embodiment; A flowchart showing an example of correction processing according to an embodiment.

[0019] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. The drawings are used for explanation purposes, and the shape, size, and size ratio of each component in an actual device do not necessarily have to be the same as those shown in the drawings. Furthermore, since the drawings are simplified, components necessary for implementation other than those shown in the drawings are also assumed to be appropriately provided.

[0020] (First embodiment)

[0021] 1 is a block diagram schematically illustrating an example of a solid-state imaging device according to an embodiment. The solid-state imaging device 10 includes an imaging unit 100, a control unit 102, a storage unit 104, a signal processing unit 106, an image processing unit 108, and an I / F 110. The solid-state imaging device 10 may also include other components (not shown) that are necessary to drive the imaging unit 100. The solid-state imaging device 10 performs flicker correction on images captured using a rolling shutter system and outputs the images.

[0022] The imaging unit 100 acquires image information using a rolling shutter system. The imaging unit 100 includes, for example, a pixel array in which pixels each having a light receiving element are arranged in an array, an optical system that focuses incident light onto the pixel array, and a pixel circuit that processes and outputs signals acquired by the light receiving elements. The imaging unit 100 may further include an ADC (Analog to Digital Converter) that converts analog signals into digital signals.

[0023] The imaging unit 100 outputs a digital image signal based on the acquired signal. The imaging unit 100 acquires frame images spanning multiple frames based on, for example, a set shutter timing or an exposure time (described later). The imaging unit 100 acquires a moving image or still image with flicker correction from the multiple frame images. The shutter timing and exposure time indicate the timing at which the frame images are acquired.

[0024] The control unit 102 is a circuit that controls each component of the solid-state imaging device 10. The control unit 102 can perform control based on, for example, control generated in the signal processing unit 106, the image processing unit 108, etc., or control based on an external request via the I / F 110. Each component of the solid-state imaging device 10 may be controlled based on a signal from the control unit 102.

[0025] The storage unit 104 is a storage area that can store data required for processing in the solid-state imaging device 10, image data after processing, etc. The storage unit 104 can include, for example, various RAMs (Random Access Memories), ROMs (Read Only Memories), etc. The storage unit 104 may also include a register, as indicated by the dotted line, that can temporarily store signals acquired via the I / F 110 or non-temporarily store various setting values.

[0026] The signal processing unit 106 is a circuit that performs signal processing or image processing on the digital image signal output by the imaging unit 100. The control unit 102 performs general digital signal processing in the solid-state imaging device 10, such as defect correction processing, filter processing, and linear matrix processing, and can also perform correction processing to reduce flicker.

[0027] The signal processing unit 106, for example, performs detection on blocks of images from multiple frame images acquired by the imaging unit 100, estimates the flicker period occurring in the blocks, and outputs the result. As a specific, non-limiting example, the signal processing unit 106 can estimate the flicker period by calculating the autocorrelation value of the detection values ​​in different frame images. The signal processing unit 106 can also obtain the spatial period of flicker stripes (in-plane flicker) present in the images based on the detection values ​​for each block, and perform control to suppress in-plane flicker based on this.

[0028] The image processing unit 108 can perform image processing on the resulting digital image signal output by the imaging unit 100 (or the signal processing unit 106). The image processing unit 108 can perform processing to generate an image with reduced in-screen flicker, for example, by performing gain correction for each block and recombining the blocks that have undergone gain correction.

[0029] It is not necessary to strictly separate the operations of the signal processing unit 106 and the image processing unit 108, and they can be integrated into a processing circuit, or each processing unit can be formed as a single processing circuit or a set of multiple processing circuits. At least a part of the processing circuit may be a general-purpose processor, or at least a part may be formed by a dedicated circuit formed by an ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), etc.

[0030] The I / F 110 is an interface that connects the inside and outside of the solid-state imaging device 10. The I / F 110 may include an interface for transmitting and receiving data, or may include a user interface that can present an acquired image to a user and input requests from the user.

[0031] Figure 2 shows an example of how in-plane flicker occurs. As shown in this figure, in-plane flicker can appear as a striped pattern in an image. When capturing an image using the rolling shutter method, pixels on the same line are generally exposed at the same time, and scanning is performed with a time difference in the column direction, shifting the exposure time, resulting in a striped pattern in the column direction.

[0032] In this disclosure, the general terms line and column are used, but these are not limited to these and can be interpreted as the direction in which the exposure timing (shutter timing) coincides and the direction in which the exposure timing (shutter timing) is scanned, respectively.

[0033] The solid-state imaging device 10 divides an image into blocks that include pixels that belong to the same line, i.e., pixels that have the same exposure timing, and calculates parameters for correcting flicker based on the luminance values ​​of these blocks in multiple frame images acquired in chronological order, thereby correcting flicker in the image. This processing can be performed, for example, in the signal processing unit 106 or the image processing unit 108.

[0034] The solid-state imaging device 10 divides an image into blocks, for example, with a plurality of consecutive lines as one block. The image can be divided into a predetermined number of blocks, a predetermined number of lines can be specified, or the number of blocks can be automatically set based on the width of the flicker.

[0035] When dividing by a predetermined number of blocks, the solid-state imaging device 10 divides the total number of lines by the predetermined number of blocks, sets the number of lines based on the integer value of the quotient (e.g., a value obtained by truncating or rounding off the decimal point), and divides the frame image for each of the set lines. For the last block, the solid-state imaging device 10 can treat the remaining lines as one block. The predetermined number of blocks can be, for example, 30 blocks, but is not limited to this.

[0036] When dividing by a predetermined number of lines, the solid-state imaging device 10 can set blocks for each set predetermined number of lines. The predetermined number of lines can be, for example, 10 lines, but is not limited to this. It is desirable that the predetermined number of lines has a certain amount to ensure robustness against vibrations of the imaging unit 100, vibrations of the object to be imaged, etc.

[0037] When controlling the number of blocks based on the flicker width, for example, the width of the in-plane flicker can be estimated using a method described below, and then the number of lines in a block can be determined based on the value obtained by dividing the flicker width by a predetermined number, and the image can be divided into blocks.

[0038] 3 is a diagram showing an example of dividing an image into blocks. The solid-state imaging device 10 divides the image as shown by the dotted lines to generate multiple blocks. This division is performed for each frame image acquired in chronological order.

[0039] The solid-state imaging device 10 acquires parameters for correcting flicker based on, for example, the luminance values ​​of each block in consecutive frames.

[0040] That is, the signal processing unit 106 and / or the image processing unit 108, which are processing circuits of the imaging unit 100, generate blocks of multiple lines for multiple images acquired in a rolling shutter format that are consecutive in time series, estimate the period of time-dependent flicker caused by the light source from the luminance values ​​in these blocks, and perform processing to suppress flicker caused by the light source based on at least this frame period.

[0041] 4 is a flowchart showing an in-screen flicker correction process according to an embodiment. This correction process corrects the gain, and by correcting the gain for each block, in-screen flicker of an image is suppressed.

[0042] The processing circuit divides the image region into blocks (S100). The blocks are generated by dividing the image region based on appropriate conditions as described above.

[0043] The processing circuit estimates the frame period of the flicker (S102). The frame period of the flicker is a period synchronized with the blinking period of the light source and indicates a period in the time series direction. The processing circuit, for example, extracts an arbitrary block, obtains the luminance value of the entire block, and estimates the frame period of the flicker from changes in the luminance value of the entire block. The processing circuit can, for example, determine the luminance value of the block as the sum or average value of the luminance values ​​of the pixels belonging to the block. Hereinafter, when the luminance value of a block is referred to, the same value is indicated.

[0044] 5 is a diagram illustrating calculation of correlation values ​​of luminance values ​​of blocks between frames according to an embodiment. For example, the processing circuit calculates autocorrelation using luminance values ​​of blocks for seven frames, and obtains the frame shift at which the autocorrelation value is maximum as the flicker period. For example, the processing circuit obtains luminance values ​​of blocks in consecutive frames, and for a luminance value sequence x in a certain frame, obtains luminance values ​​of a luminance value sequence shifted by one frame, a luminance value sequence shifted by two frames, and so on, and obtains an autocorrelation value r based on the following equation:

[0045] where s xy is the covariance of x and y, sx is the standard deviation of x, s y is the standard deviation of y, n is the total number of bivariate data (x, y), x i and y i are the luminance values ​​in the luminance value sequence of x and y, respectively, and x bar and y bar represent the average luminance values ​​of the blocks in the luminance value sequence of x and y, respectively. y indicates the luminance value sequence in a frame shifted by m frames from x. m and n can be set appropriately depending on the processing capacity of the processing circuit. For example, m = 10, n = 7, etc. can be used, but are not limited to these.

[0046] The processing circuit estimates the frame period of the flicker based on the autocorrelation value r shown in equation (1) from the luminance values ​​of the block. In the example of Figure 5, the processing circuit can determine that the autocorrelation value with the luminance value sequence shifted by three frames is the maximum, and can estimate these three frames as the frame period of the flicker.

[0047] The processing circuit can calculate the frame period of the flicker from the maximum value of the autocorrelation value along the time series in any of a plurality of blocks, and can also estimate the frame period of the flicker for the entire image based on the calculated frame periods of the plurality of flicker.

[0048] By extracting blocks and estimating the frame period of flicker, it is possible to estimate the frame period of high-speed flicker.

[0049] The processes from S100 to S102 may be a flicker frame period estimation process (S10).

[0050] After estimating the frame period of the flicker, the processing circuit calculates a correction gain value for each block (S200).

[0051] Fig. 6 is a diagram schematically illustrating calculation of a correction gain value according to an embodiment. Fig. 6 is a graph illustrating changes in the luminance value of a certain block. The processing circuit calculates a correction gain value such that the luminance value of the block in a frame of interest becomes Imax, where Imax is the maximum luminance of a certain flicker in a frame period.

[0052] For example, the processing circuit can calculate a correction gain value g0 for correcting the luminance value of a block for an image of a frame at time t0 as g0 = Imax / I0. Similarly, the processing circuit can calculate a correction gain value g2 for correcting the luminance value of a block for an image of a frame at time t2 as g2 = Imax / I2.

[0053] The processing circuit calculates a correction gain value for each block in the frame to be corrected.

[0054] The processing circuit performs gain correction for each pixel belonging to each block based on the correction gain value calculated for each block (S202). The processing circuit multiplies the luminance value of each pixel belonging to the block by the correction gain value calculated in S200 to obtain a gain-corrected luminance value.

[0055] The processing circuit combines the blocks that have been gain-corrected to obtain a gain-corrected frame image (S204). This process enables the solid-state imaging device 10 to obtain a gain-corrected, flicker-suppressed frame image.

[0056] 7 is a diagram schematically illustrating gain correction of a frame image. As shown in the diagram, the processing circuit calculates a correction gain value for each block, performs gain correction on the block using the correction gain value, and combines each of the gain-corrected blocks to obtain a frame image in which flicker is suppressed.

[0057] As described above, according to this embodiment, the solid-state imaging device 10 divides a frame image into blocks and can quickly perform calculations to obtain the frame period of flicker by using any block. Furthermore, the solid-state imaging device 10 can quickly correct in-plane flicker by performing gain correction based on the frame period of flicker obtained for each block.

[0058] (Second embodiment)

[0059] In the first embodiment, flicker is suppressed by gain correction, but the suppression of in-plane flicker in the present disclosure is not limited to this method. The solid-state imaging device 10 can also suppress in-plane flicker by controlling, for example, the shutter timing (including the exposure time). Hereinafter, when the term "shutter value" is used, this is a concept that includes at least the exposure time, and may also be a concept that includes the timing at which exposure starts.

[0060] In this embodiment, the solid-state imaging device 10 calculates a corrected shutter value using at least the frame period of flicker estimated from blocks in a frame image by the same process as described above. The solid-state imaging device 10 calculates the corrected shutter value in a processing circuit and outputs this shutter value to the control unit 102, thereby controlling the shutter timing in the imaging unit 100.

[0061] FIG. 8 is a flowchart showing an example of a correction process for correcting the shutter value according to an embodiment.

[0062] The processing circuit divides the frame image into blocks, as in the previous embodiment, and estimates the frame period of the flicker using information from any block (S10).

[0063] The processing circuit estimates the light source frequency based on at least the frame period of the flicker (S30).

[0064] The processing circuit calculates a corrected shutter value based on the light source frequency estimated in S30 (S400). The processing circuit calculates, for example, an exposure time for a frame as the corrected shutter value. Furthermore, the processing circuit may calculate the exposure start timing as the corrected shutter value.

[0065] The processing circuit outputs the corrected shutter value calculated in S400 to the control unit 102, and the control unit 102 controls the shutter of the imaging unit 100 based on the obtained corrected shutter value (S402). As a result, the solid-state imaging device 10 can obtain frame images with suppressed in-plane flicker.

[0066] As shown in the flowchart, after the process of S30, the solid-state imaging device 10 performs a process of calculating a corrected shutter value and controlling the shutter (S40), thereby capturing an image in which in-plane flicker is suppressed.

[0067] The processing of S30 will be explained in more detail. If the light source can be limited to a fluorescent lamp, the processing of S30 is equivalent to estimating the blinking period of a fluorescent lamp. On the other hand, if there is a possibility that the light source may include an LED, it is necessary to estimate the blinking period of the LED.

[0068] FIG. 9 is a flowchart showing an example of the above-mentioned light source frequency estimation process (S30).

[0069] The processing circuit determines whether the light source is to be specialized as a fluorescent lamp (S300). This determination can be made, for example, based on information previously requested by the user via the I / F 110 or the like. As another example, the processing circuit can detect whether the light source is a fluorescent lamp or an LED using some method and make the determination based on this detection result. This method can be any method that appropriately distinguishes between a fluorescent lamp and an LED. For example, fluctuations in the light source intensity can be acquired and the determination can be made based on the frequency of these fluctuations. As a non-limiting example, an LED light source can be set as the default light source, or the user can switch to a fluorescent lamp. As a non-limiting example, a configuration is possible in which the processing of S300 is omitted, assuming that the processing of S304 and S306 is performed.

[0070] When performing correction specific to fluorescent lights (S300: YES), the processing circuit identifies the light source frequency that causes the flicker from the inter-plane flicker period (flicker frame period) in an arbitrary block and the image capture frame rate (S302). Fluorescent lights are often expressed with a limited flicker period, such as 100 Hz or 120 Hz, depending on regional conditions, etc.

[0071] The processing circuit can identify the light source frequency by setting the imaging frame rate (FPS: Frames Per Second) to 24, 25, 30, 50, 60, etc., and obtaining estimated values ​​of the flicker frame period at these FPS. For example, if the flicker frame period is 6 frames at 24 FPS, 3 frames at 30 FPS, and 3 frames at 60 FPS, the processing circuit can identify the light source frequency as 100 Hz. Also, if the flicker frame period is 5 frames at 25 FPS and 5 frames at 50 FPS, the processing circuit can identify the light source frequency f s can be estimated to be 120Hz.

[0072] As another example, the processing circuit may acquire location information of the solid-state imaging device 10, and based on this location information, set the power frequency of the area where the solid-state imaging device 10 is located as the initial value for estimating the light source frequency. In this case, the above processing may also be executed to further determine whether there is consistency.

[0073] On the other hand, if the process is not specialized for fluorescent lamps (S300: NO), the processing circuit proceeds to a process of estimating the in-plane flicker period (S304). This process can be applied to both fluorescent lamps and LEDs.

[0074] As an example, the processing circuit can calculate a correction gain value for each block in the same way as in the first embodiment, and obtain the intra-screen flicker period based on this correction gain value.

[0075] 10 is a diagram showing a correction gain value for each block in a frame image according to an embodiment, and a diagram showing a process for calculating an in-screen flicker period. The processing circuit calculates the in-screen flicker period based on the correction gain value calculated for each block.

[0076] The processing circuit calculates the correction gain values ​​by the same process as in the above-described embodiment, for example, as shown on the right side of Fig. 10. Since the correction gain values ​​correspond to the luminance values ​​of the blocks, a higher correction gain value is set for a dark block than for a lighter block, and a lower correction gain value is set for a light block than for a darker block, as shown in the figure.

[0077] As a result, the in-plane flicker period can be calculated by referring to the correction gain value. The processing circuit can calculate the in-plane flicker period by performing a calculation in the spatial direction (for example, the column direction in which flicker stripes occur) that is equivalent to the calculation process of the autocorrelation value in the time direction in the first embodiment.

[0078] Specifically, by shifting the correction gain value sequence in a certain frame in the block direction and calculating the autocorrelation value in order, the shift in the number of blocks at which the autocorrelation value becomes maximum can be searched for, and this search result can be obtained as the intra-screen flicker period. In this case, the intra-screen flicker period is calculated in units of blocks.

[0079] Although the in-plane flicker period is calculated using the correction gain value in the above example, the present invention is not limited to this. For example, the processing circuit can perform an autocorrelation calculation in the same block division direction using the luminance value of the block instead of the correction gain value, and calculate the in-plane flicker period from the result of this autocorrelation calculation.

[0080] Furthermore, according to the above, the in-plane flicker period is basically obtained as an integer value, but the processing circuit can also interpolate correction gain values ​​around the deviation in the number of blocks at which the autocorrelation value is maximized, as shown by the dotted line in FIG. 10, and use the interpolated curve to calculate the in-plane flicker period after the decimal point.

[0081] After estimating the in-screen flicker cycle, the processing circuit calculates the light source frequency (S306). The processing circuit calculates the light source frequency based on, for example, the following equation.

[0082] where f s denotes the light source frequency, and pspatial denotes the in-plane flicker period, and τ denotes the readout time per block. According to equation (2), the block-based light source frequency f s can be obtained.

[0083] However, this calculation result may have errors depending on the number of divisions into blocks (i.e., the number of lines constituting the block). For this reason, the processing circuit may further perform processing to improve the accuracy of the result calculated in equation (2). For example, the processing circuit can obtain candidate values ​​for the light source frequency using an integer N that is relatively prime to the frame period of the flicker, as shown in the following equation:

[0084] where f' s is the candidate value of the light source frequency, r frame is the image capture frame rate of the image capture unit 100, p frame indicates the frame period of the flicker. The processing circuit calculates candidate values ​​for the light source frequency from the imaging frame rate and the frame period of the flicker. For example, if the imaging frame rate is 100 fps and the frame period of the flicker is 2 frames, the candidate values ​​for the light source frequency are [50 Hz, 150 Hz, 250 Hz, 350 Hz, ...].

[0085] The processing circuit can use the candidate values ​​calculated by equation (3) to improve the accuracy of the value calculated by equation (2). For example, the candidate light source frequencies are calculated by equation (3) to be [50 Hz, 150 Hz, 250 Hz, 350 Hz, ...], and the f calculated by equation (2) can be used. s is 240 Hz, the processing circuitry can obtain the closest candidate value to 240 Hz, 250 Hz, as the light source frequency.

[0086] After these processes (S302 or S306), the process of calculating the corrected shutter value (S400) shown in Fig. 8 is executed. The processing circuit calculates the corrected exposure time as the corrected shutter value using, for example, the following formula.

[0087] where t shutterindicates the exposure time calculated as the corrected shutter value. By setting the correction time in this way, it is possible to set the exposure time for each line to be equal to the blinking period of the light source or an integer multiple of the blinking period.

[0088] It is desirable to set M within a range where M times the flickering periphery of the light source is contained in one cycle of the imaging frame rate. M may also be set depending on the environment or user requirements, such as whether a high shutter speed (M: small) is desirable to avoid motion blur or a sufficiently long shutter speed (M: large) is desirable in a dark environment.

[0089] The processing circuit transmits the calculated exposure time to the control unit 102, and the control unit 102 controls the exposure time in the imaging unit 100, thereby enabling the solid-state imaging device 10 to capture images with suppressed in-plane flicker.

[0090] FIG. 11 is a diagram showing a schematic diagram of exposure time correction. The shaded area represents the time when no exposure occurs. The dotted line represents the brightness of the light source. Before correction, the exposure time for each line is set based on the image capture frame rate for this light source brightness. The timing of this exposure time causes in-screen flicker to occur on the screen, depending on the flicker frame period and the image capture frame rate.

[0091] Once the exposure time calculated by the above process is set, the imaging unit 100 performs imaging for each line using an exposure time equivalent to M blinking periods (M ∈ {1, 2, 3, ...}). Therefore, an exposure time that is an integer multiple of the blinking period of the light source is set for each line, making it possible to suppress in-screen flicker that depends on the flicker frame period and the imaging frame rate.

[0092] As described above, according to this embodiment, in-screen flicker can be appropriately suppressed by automatically setting the exposure time. This exposure time can be set by selecting either a method specialized for fluorescent lamps or a method compatible with both fluorescent lamps and LEDs.

[0093] (Third embodiment)

[0094] In the above-described embodiments, gain correction and shutter value correction are set separately, but it is also possible to configure these to be selectable.

[0095] 12 is a flowchart showing an example of the correction process according to an embodiment. The solid-state imaging device 10 acquires settings related to the correction process and writes them to the storage unit 104 (S500). The solid-state imaging device 10 acquires a request from a user via the I / F 110, for example, and writes data indicating a correction method according to the request to a register.

[0096] The solid-state imaging device 10 captures an image in the imaging unit 100 and acquires image data (S502). The solid-state imaging device 10 can acquire images of consecutive frames in a time series based on parameters such as a shutter value that are set as initial values.

[0097] After this, the solid-state imaging device 10 executes the process of S10.

[0098] After the process of S10 is completed, the solid-state imaging device 10 reads the value written in the register to determine whether or not to perform gain correction (whether execution of gain correction is ON or OFF) (S504).

[0099] If the gain correction is ON (S504: YES), the solid-state imaging device 10 executes the process of S20.

[0100] If the gain correction is not ON (S504: NO), the solid-state imaging device 10 executes the processes of S30 and S40. In the process of S30, the determination process of S300, like the process of S504, can refer to the value written in the register to determine whether to execute processing specialized for fluorescent lamps or processing compatible with LEDs.

[0101] After the process of S20 or S40 is completed, the solid-state imaging device 10 outputs the corrected image (S506) and completes the process. The output is not limited to an image. For example, it may also output information such as whether or not in-plane flicker exists, or the light source frequency.

[0102] In the above description, the selection is made based on the request of the user, etc. However, the present invention is not limited to this. For example, the solid-state imaging device 10 can automatically determine whether to perform gain correction or shutter value correction internally or externally to the solid-state imaging device 10, and execute flicker stripe suppression processing based on this determination.

[0103] As described above, according to the present disclosure, it is possible to estimate the light source frequency in a frame image captured using a rolling shutter method without requiring a sensor dedicated to flicker detection. Furthermore, it is not necessary to search for the light source frequency by changing the exposure time multiple times, and the light source frequency can be estimated efficiently. In these processes, parameters can be calculated on a block-by-block basis, rather than on an image-by-image basis, enabling near-real-time processing. Furthermore, when performing gain correction, flicker correction can be performed without estimating the light source frequency.

[0104] Because the light source frequency can be automatically searched for, it is possible to suppress flicker caused by light sources such as LEDs, whose light source frequency is not uniquely determined. For shutter value correction, it is possible to select between fluorescent light specialized mode and fluorescent light / LED compatible mode, allowing appropriate flicker correction to be performed depending on the scene.

[0105] By selecting the above correction, users who are concerned about restrictions on exposure time due to shaking, motion blur, etc. can select gain correction, and users who are concerned about increased noise sensitivity due to increased gain can select shutter correction.

[0106] The above-described embodiment may be modified as follows.

[0107] (1) A solid-state imaging device comprising: a processing circuit, wherein the processing circuit divides a plurality of images acquired in a rolling shutter format that are consecutive in time series, in a first direction in which the shutter timing is the same and in a second direction that intersects with the first direction and scans the shutter timing, so that pixels that belong to at least the same first direction are in the same block; estimates a frame period of flicker caused by a light source from luminance values ​​in the block; and executes processing to suppress flicker caused by the light source based on at least the frame period.

[0108] (2) The solid-state imaging device according to (1), wherein the processing circuit estimates the frame period based on a change in luminance value in the block.

[0109] (3) The solid-state imaging device according to (2), wherein the processing circuit calculates an autocorrelation value of changes in luminance values ​​in the blocks acquired in a time series, and estimates the period at which the autocorrelation value is maximum as the frame period.

[0110] (4) A solid-state imaging device according to any one of (1) to (3), wherein the processing circuit calculates a correction gain value for each block in each frame so that the luminance value of the block is the maximum value of the luminance value of the block in the frame period, corrects the luminance value of each pixel in each block based on the correction gain value calculated for each block, and combines the blocks with corrected luminance values ​​within the frame to obtain an image with suppressed flicker.

[0111] (5) The solid-state imaging device according to any one of (1) to (4), wherein the processing circuit calculates an exposure time based on the frame period.

[0112] (6) The solid-state imaging device according to (5), wherein the processing circuit calculates the exposure time as a product of the frame period and a natural number.

[0113] (7) The solid-state imaging device according to (5), wherein the processing circuit further calculates an exposure time based on the correction gain value.

[0114] (8) The solid-state imaging device described in (7), wherein the processing circuit calculates an in-plane flicker period that exists across the blocks from the correction gain value for each block in an image in a frame, and calculates the exposure time based on the frame period and the in-plane flicker period.

[0115] (9) The solid-state imaging device according to (8), wherein the processing circuit estimates a light source frequency based on the frame period and the in-plane flicker period, and calculates the exposure time as a natural number multiple of the light source blinking period calculated from the light source frequency.

[0116] (10) A solid-state imaging device according to any one of (5) to (9), further comprising an imaging unit that acquires frame images in a rolling shutter format, wherein the processing circuit controls the exposure time of the imaging unit based on the calculated exposure time.

[0117] (11) The solid-state imaging device according to any one of (1) to (4), further comprising an imaging unit that acquires frame images in a rolling shutter format.

[0118] The aspects of the present disclosure are not limited to the above-described embodiments and include various conceivable modifications, and the effects of the present disclosure are not limited to the above-described contents. The components in each embodiment may be appropriately combined and applied. In other words, various additions, modifications, and partial deletions are possible within the scope of the conceptual idea and intent of the present disclosure, which is derived from the content defined in the claims and their equivalents.

[0119] 10: solid-state imaging device, 100: imaging unit, 102: control unit, 104: storage unit, 106: signal processing unit, 108: image processing unit, 110: I / F

Claims

1. A solid-state imaging device comprising: a processing circuit, which divides a plurality of images acquired in a rolling shutter format that are consecutive in time series, in a first direction in which the shutter timing is the same and in a second direction that intersects with the first direction and scans the shutter timing, so that pixels that belong to at least the same first direction are in the same block; estimates a frame period of flicker caused by a light source from luminance values ​​in the block; and executes processing to suppress flicker caused by the light source based on at least the frame period.

2. The solid-state imaging device according to claim 1, wherein the processing circuit estimates the frame period based on a change in luminance value in the block.

3. The solid-state imaging device according to claim 2, wherein the processing circuit calculates an autocorrelation value of changes in luminance values ​​in the blocks acquired in time series, and estimates the period at which the autocorrelation value is maximum as the frame period.

4. The solid-state imaging device of claim 1, wherein the processing circuit calculates a correction gain value for each block in each frame so that the luminance value of the block is the maximum value of the luminance value of the block in the frame period, corrects the luminance value of each pixel in each block based on the correction gain value calculated for each block, and combines the blocks with corrected luminance values ​​within the frame to obtain an image with suppressed flicker.

5. The solid-state imaging device according to claim 1, wherein the processing circuit calculates an exposure time based on the frame period.

6. The solid-state imaging device according to claim 5, wherein the processing circuit calculates the exposure time as a product of the frame period and a natural number.

7. The solid-state imaging device according to claim 5, wherein the processing circuit further calculates an exposure time based on the correction gain value.

8. The solid-state imaging device according to claim 7, wherein the processing circuit calculates an in-plane flicker period present across the blocks from the correction gain value for each block in an image of a frame, and calculates the exposure time based on the frame period and the in-plane flicker period.

9. The solid-state imaging device according to claim 8, wherein the processing circuit estimates a light source frequency based on the frame period and the in-plane flicker period, and calculates, as the exposure time, a natural number multiple of the light source blinking period calculated from the light source frequency.

10. The solid-state imaging device according to claim 5, further comprising an imaging unit that acquires frame images in a rolling shutter format, wherein the processing circuit controls the exposure time of the imaging unit based on the calculated exposure time.

11. The solid-state imaging device according to claim 1, further comprising an imaging section that acquires frame images in a rolling shutter format.

Images