Light sensor capable of counteracting image flicker

Abstract

A light sensor acquires a first image frame relative to a first flicker period and a second image frame relative to a second flicker period. The light sensor adds the first image frame and the second image frame to generate an image frame sum for motion detection; or the light sensor adds every two pixel data of the adjacent pixel rows of the first image frame and the second image frame to generate a low-resolution image frame for motion detection.

Description

[0001] This application is a divisional application of the Chinese invention patent application with application number 202210511330.1, application date May 11, 2022, and name “Optical sensor for offsetting image flicker”. Technical Field

[0002] The present invention relates to a light sensor, and more particularly to a light sensor capable of reducing or even eliminating image flicker by adding pixel data of two adjacent pixel rows of two image frames or one image frame. Background Art

[0003] The motion detection device using a light sensor determines the motion of an object by calculating the change in light intensity in the image frame obtained by the light sensor. However, when such a motion detection device is operated indoors, the indoor ambient light is sometimes provided by fluorescent tubes and the AC nature of the power system has flickering. Therefore, when the frame rate of the light sensor is not synchronized with the flickering frequency of the ambient light, the average brightness of the image frame obtained by the light sensor will also change, which may lead to the occurrence of erroneous judgment of the motion. The flickering frequency of the ambient light is determined by the AC frequency of the power system.

[0004] For example, refer to Figure 1 , which shows a schematic diagram of a known light sensor acquiring image frames relative to ambient light changes. When the light sensor acquires image frame 1 when the ambient light is brightest, the average brightness of image frame 1 is the highest; when the light sensor acquires image frame 2 when the ambient light is darkest, the average brightness of image frame 2 is the lowest; and the average brightness of image frame 3 is between image frame 1 and image frame 2. The intensity change from image frame 1 to image frame 3 may lead to the wrong judgment of the object movement. Summary of the invention

[0005] In view of this, the present invention further provides a light sensor that reduces or even eliminates image flicker by adding pixel data of two adjacent pixel rows of two image frames or one image frame.

[0006] The present invention provides a light sensor which adds pixel data of two image frames at the digital end to reduce or even eliminate image flicker.

[0007] The present invention also provides a light sensor which adds two pixel data of adjacent pixel rows of an image frame at the digital end to reduce or even eliminate image flicker.

[0008] The present invention also provides a light sensor which adds two pixel data of adjacent pixel rows of an image frame at the analog end to reduce or even eliminate image flicker.

[0009] The present invention provides a light sensor for acquiring image frames relative to a flickering period of ambient light. The light sensor comprises a pixel array, a reading circuit and a processor. The pixel array comprises a plurality of pixel rows for acquiring pixel data with a rolling shutter. The reading circuit is used to read the pixel data of the plurality of pixel rows respectively. The processor is used to add the pixel data of the Nth row of pixels and the N+1th row of pixels of the plurality of pixel rows respectively to generate a first output image, wherein N is a positive odd number starting from 1.

[0010] The present invention also provides a light sensor for acquiring image frames relative to the flickering cycle of ambient light. The light sensor includes a pixel array, a reading circuit and a processor. The pixel array includes a plurality of pixel rows for acquiring pixel data with a rolling shutter. The reading circuit is used to read the data sum of the pixel data of the Nth row of pixels and the N+1th row of pixels of the plurality of pixel rows, respectively, wherein N is a positive odd number starting from 1. The processor is used to generate an output image based on the plurality of data sums, wherein the vertical resolution of the output image is half the number of the plurality of pixel rows.

[0011] In order to make the above and other purposes, features and advantages of the present invention more obvious, the following will be described in detail with reference to the accompanying drawings. In addition, in the description of the present invention, the same components are represented by the same symbols, which are hereby described together. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a schematic diagram of obtaining an image frame by a known light sensor relative to ambient light changes;

[0013] Figure 2 is a block diagram of a light sensor according to an embodiment of the present invention;

[0014] Figure 3A and Figure 3B is a schematic diagram of an operating method of an optical sensor according to a first embodiment of the present invention;

[0015] Figure 4A and Figure 4B is a schematic diagram of an operating method of an optical sensor according to a second embodiment of the present invention;

[0016] Figure 5A is a schematic diagram of an operating method of an optical sensor according to a third embodiment of the present invention;

[0017] Figure 5B A pixel circuit of a photosensor according to a third embodiment of the present invention; and

[0018] Figure 5C yes Figure 5B The switching state of the pixel circuit during operation.

[0019] Description of Reference Numerals

[0020] 200 Light Sensor

[0021] 21 Pixel Array

[0022] 23 Reading Circuit

[0023] 25 Processor

[0024] PD1, PD2 Photodiode

[0025] T1, T2 transfer transistors

[0026] T3 Reset transistor

[0027] T4 Select transistor

[0028] T5 Source Follower Transistor

[0029] Rd Read Line DETAILED DESCRIPTION

[0030] The optical sensor of the embodiment of the present invention is suitable for eliminating the influence of light intensity variation when acquiring image frames under time-varying ambient light with a flickering cycle, thereby increasing the accuracy of control according to image processing results.

[0031] Please refer to Figure 2 , which is a block diagram of an optical sensor 200 according to an embodiment of the present invention. The optical sensor 200 is, for example, a CMOS image sensor or a CCD image sensor, which can be built into various cameras or video cameras for motion detection or navigation.

[0032] The optical sensor 200 includes a pixel array 21 , a readout circuit 23 , and a processor 25 .

[0033] The pixel array 21 includes rolling shutter pixels arranged in a matrix for outputting pixel data at a frame rate to generate an image frame. In one embodiment, each pixel of the pixel array 21 includes, for example, a 4T pixel circuit, for example, Figure 5B Transistors T1, T3, T4 and T5.

[0034] The readout circuit 23 is connected to the pixel array 21 and is used to read the pixel array 21 through a plurality of readout lines (eg Figure 5B The readout circuit 23 reads out the pixel data of each pixel of the pixel array 21. For example, the readout circuit 23 reads each pixel row of the pixel array 21 in sequence according to the row selection signal. The row selection signal is generated by a timing controller, for example. The operation of the timing controller is known and is not the main purpose of the present invention, so it will not be described in detail here.

[0035] The processor 25 is, for example, an application specific integrated circuit (ASIC), a digital signal processor (DSP) or a programmable logic circuit (FPGA), which receives and processes the pixel data read by the reading circuit 23, such as calculating the pixel data and (examples are described below).

[0036] Please refer to Figure 3A and Figure 3B , which is a schematic diagram of the operation method of the light sensor 200 according to the first embodiment of the present invention. The first embodiment of the present invention cancels the image flicker by calculating the sum of two image frames, for example, adding the pixel data of the pixels at corresponding positions of the two image frames respectively.

[0037] The pixel array 21 is used to obtain a first image frame relative to a first flickering period, and to obtain a second image frame relative to a second flickering period. Figure 3A and Figure 3B The first image frame and the second image frame in FIG. 2 only display three pixel rows Row1 to Row3 , but the present invention is not limited thereto. The number of pixel rows depends on the actual size of the pixel array 21 .

[0038] For example, Figure 3A and Figure 3B In FIG. 1 , the start time of the first image frame is shown as T1 and the start time of the second image frame is shown as T2. The first image frame and the second image frame are respectively acquired by a rolling shutter, so Figure 3A and Figure 3B There is a time difference between the pixel rows (eg, the difference between the start time of pixel rows Row1 and Row2, the difference between the start time of pixel rows Row2 and Row3, etc.) The method of acquiring image frames with a rolling shutter is well known, so it will not be described here.

[0039] The reading circuit 23 is used to read the first image frame and the second image frame during different frame periods. Figure 3A and Figure 3B The readout period for each pixel row is shown after the exposure period.

[0040] The processor 25 is used to add the first image frame and the second image frame (e.g., add corresponding pixels of the two image frames) to generate an image frame sum to offset the image flicker caused by ambient light, that is, the image frame sum has the same size as the first image frame and the second image frame. In the present invention, the image frame sum is provided to a digital backend (e.g., a host) for image recognition, motion detection, storage, playback, etc., depending on different applications.

[0041] In the first embodiment, the exposure period of each pixel row of the first image frame and the second image frame is preferably less than the flicker period of the ambient light. Figure 3AThe exposure period of each pixel row for displaying the first image frame and the second image frame is 1 / 2 of the flickering period (e.g., 1 / 240 seconds); Figure 3B The exposure period of each pixel row displaying the first image frame and the second image frame is 1 / 4 of the flickering period (eg, 1 / 480 seconds), but the present invention is not limited thereto.

[0042] It must be pointed out that although Figure 3A and Figure 3B The display flicker cycle is 1 / 120 second, but the present invention is not limited thereto. The value of the flicker cycle is determined according to the AC frequency of the power system in the area where the optical sensor 200 is located, for example, in some areas it is 1 / 100 second, but the present invention is not limited thereto.

[0043] It must be pointed out that although Figure 3A and Figure 3B The first image frame and the second image frame are displayed 1.5 times the flicker period (i.e., the frame period is 1.5 times the flicker period) to completely eliminate image flicker, but the present invention is not limited thereto. As long as the first image frame and the second image frame are 1.5 to 2 times the flicker period, the image flicker can be reduced.

[0044] In addition, although Figure 3A and Figure 3B The start time T1 of the first image frame is aligned with the zero phase (or start phase) of the flicker cycle, but it is only used for illustration and not for limiting the present invention. As long as the first image frame and the second image frame are separated by 1.5 to 2 times of the flicker cycle, no matter the start time T1 is located at any phase of the flicker cycle, the image flicker can be reduced or even eliminated by adding the first image frame and the second image frame (for example, adding the corresponding pixels of the two image frames).

[0045] It must be pointed out that although Figure 3A and Figure 3B Two adjacent flickering periods of the first image frame and the second image frame relative to the ambient light are displayed, but the present invention is not limited thereto. In other embodiments, the first image frame and the second image frame may be separated by more than two flickering periods (i.e., the frame period is more than twice the flickering period), for example, between 2.5 times and 3 times, between 3.5 times and 4 times, and so on, and image flicker can also be reduced or even eliminated by adding two adjacent image frames.

[0046] Please refer to Figure 4A and Figure 4B , which is a schematic diagram of the operation method of the light sensor 200 according to the second embodiment of the present invention. The second embodiment of the present invention is to cancel the image flicker by calculating the sum of pixel data of corresponding pixel positions of two adjacent pixel rows.

[0047] Similarly, the pixel array 21 includes a plurality of pixel rows to acquire pixel data using a rolling shutter.

[0048] The reading circuit 23 is used to read the pixel data of the plurality of pixel rows respectively, for example Figure 4A and Figure 4B The readout period for each pixel row is shown after the exposure period.

[0049] The processor 25 is used to add the pixel data of the Nth row of pixels and the N+1th row of pixels of the plurality of pixel rows respectively (for example, adding the pixels located in the same column in the adjacent Nth row and the N+1th row) to generate a first output image, wherein N is a positive odd number starting from 1, such as 1, 3, 5, etc., and the final value of N is determined according to the size of the pixel array 21. For example, the processor 25 adds the pixel data of the first row of pixels acquired by the pixel array 21 and the pixel data of the second row of pixels to form the pixel data of the first row of pixels of the first output image; adds the pixel data of the third row of pixels acquired by the pixel array 21 and the pixel data of the fourth row of pixels to form the pixel data of the second row of pixels of the first output image; and so on. In this embodiment, the processor 25 does not add the pixel data of the second row of pixels to the pixel data of the third row of pixels; does not add the pixel data of the fourth row of pixels to the pixel data of the fifth row of pixels; and so on. Therefore, the vertical resolution (i.e., the number of pixel rows) of the first output image is half the number of the plurality of pixel rows of the pixel array 21.

[0050] from Figure 4A and Figure 4B It can be seen that during the exposure period of the first row of pixels of the pixel array 21, the ambient light gradually increases, while during the exposure period of the second row of pixels of the pixel array 21, the ambient light gradually decreases. During the exposure period of the third row of pixels of the pixel array 21, the change trend of the ambient light is opposite to the change trend of the ambient light during the exposure period of the fourth row of pixels of the pixel array 21.

[0051] Therefore, the results of adding the pixel data of the first row of pixels and the second row of pixels acquired by the pixel array 21 in pairs and adding the pixel data of the third row of pixels and the fourth row of pixels in pairs are substantially the same (ie Figure 4B The integral areas of the oblique lines in different directions are the same as shown in the figure), so the image flicker can be eliminated.

[0052] Although Figure 4A and Figure 4BIn the display pixel array 21, the time difference between the exposure start time of the N+1th row of pixels (e.g., t2, t4) and the exposure start time of the Nth row of pixels (e.g., t1, t3) is 1 / 2 of the flicker cycle, but the present invention is not limited thereto. In other embodiments, as long as the time difference between the exposure start time of the N+1th row of pixels of the pixel array 21 and the exposure start time of the Nth row of pixels of the pixel array 21 is between 1 / 2 and 3 / 4 of the flicker cycle, the image flicker can be reduced.

[0053] In this embodiment, the exposure period of the plurality of pixel rows of the pixel array 21 is shorter than the flickering period, preferably less than or equal to half of the flickering period. Figure 4A The exposure period for displaying the plurality of pixel rows is 1 / 2 of the flashing period (for example, 1 / 120 seconds, but not limited to); and Figure 4B An exposure period for displaying the plurality of pixel rows is 1 / 4 of the flashing period, wherein all pixels in one pixel row have the same exposure period.

[0054] In addition, if Figure 4A and Figure 4B As shown, the exposure start time of the pixels in the N+1th row (e.g., even-numbered row) of the plurality of pixel rows of the pixel array 21 is later than the exposure start time of the next pixel row. For example, the third exposure start time t3 of the pixels in the third row of the plurality of pixel rows is later than the first exposure start time t1 of the pixels in the first row of the plurality of pixel rows and earlier than the second exposure start time t2 of the pixels in the second row of the plurality of pixel rows. For example, the exposure start times (e.g., t2 and t4) of the pixels in the N+1th row of the plurality of pixel rows differ by one readout period, while the exposure start times (e.g., t1 and t3) of the pixels in the Nth row of the plurality of pixel rows differ by one readout period.

[0055] As described above, the resolution of the first output image generated by the processor 25 is relatively low. Therefore, the processor 25 preferably generates the first output image in a power saving mode (e.g., a mode when the light sensor 200 does not detect an object). In a normal mode (e.g., a mode when the light sensor 200 detects an object or receives a signal indicating that an object has been detected), the processor 25 may add the pixel data of the Mth row of pixels and the M+1th row of pixels of the plurality of pixel rows of the pixel array 21 in pairs to generate a second output image, wherein M is a positive integer starting from 1, such as 1, 2, 3, ..., and the final value of M is determined according to the size of the pixel array 21. For example, the processor 25 adds the pixel data of the first row of pixels of the pixel array 21 and the pixel data of the second row of pixels located in the same column in pairs to obtain the pixel data of the first row of pixels of the second output image; adds the pixel data of the second row of pixels of the pixel array 21 and the pixel data of the third row of pixels located in the same column in pairs to obtain the pixel data of the second row of pixels of the second output image; adds the pixel data of the third row of pixels of the pixel array 21 and the pixel data of the fourth row of pixels located in the same column in pairs to obtain the pixel data of the third row of pixels of the second output image; and so on.

[0056] In more detail, the processor 25 is further configured to determine whether an object image is acquired based on the first output image. When the processor 25 determines that the object image is not acquired, the first output image is continuously generated to the back end; when the processor 25 determines that the object image is acquired, the second output image is generated to the back end instead. The longitudinal resolution of the second output image is less than the number of the plurality of pixel rows of the pixel array 21 but greater than the longitudinal resolution of the first output image.

[0057] Please refer to Figure 5A , which is a schematic diagram of the operation method of the optical sensor 200 according to the third embodiment of the present invention. The third embodiment of the present invention also calculates the sum of pixel data of two pixels in adjacent pixel rows to cancel image flicker. The difference between the third embodiment and the second embodiment is that the processor 25 calculates the sum of pixel data outside the pixel circuit in the second embodiment, while the third embodiment calculates the sum of pixel data inside the pixel circuit, that is, it is not calculated by the processor 25.

[0058] Similarly, the pixel array 21 includes a plurality of pixel rows for acquiring pixel data using a rolling shutter.

[0059] like Figure 5BAs shown, each of the pixels in the Nth row (e.g., odd-numbered rows) of the pixel array 21 is a 4T pixel circuit, including transistors T1, T3, T4, and T5; each of the pixels in the N+1th row (e.g., even-numbered rows) is also a 4T pixel circuit, including transistors T2, T3, T4, and T5, that is, the Nth row and the N+1th row of pixels share transistors T3, T4, and T5. The photodiode PD1 of each of the pixels in the Nth row and the photodiode PD2 of the adjacent pixel in the N+1th row of pixels are connected to the same floating diffusion node FD, and the floating diffusion node FD is used to store the data sum of the pixel data of the Nth row of pixels and the N+1th row of pixels. At the same time, each of the pixels in the Nth row and the adjacent pixel in the N+1th row of pixels are connected to the same read line Rd. Figure 5B PD1 and PD2 shown in FIG. 2 are photodiodes in two adjacent rows (ie, the same column) in the vertical direction of the pixel array 21 .

[0060] Figure 5B In the embodiment, transistors T1 and T2 are used as transfer transistors. Transistor T3 is used as a reset transistor. Transistor T4 is used as a row selection transistor, which is controlled by a row selection signal Rsel. Transistor T5 is used as a source follower transistor.

[0061] Since the pixel data sum of the same pixel column is directly calculated in the pixel circuit, the reading circuit 23 is used to directly read the data sum of the pixel data of the Nth row of pixels and the N+1th row of pixels of multiple pixel rows of the pixel array 21 through multiple reading lines Rd, where N is a positive odd number starting from 1, such as 1, 3, 5...

[0062] Figure 5C The operation states of the transistors T1 to T3 of the Nth row of pixels and the N+1th row of pixels are displayed, including sequentially executing the stages of resetting PD1, exposing PD1, resetting PD2, transferring PD1, exposing PD2, and transferring PD2. Figure 5C It can be seen that the charges generated by the photodiode PD1 and the photodiode PD2 are sequentially transferred through the charge transfer (i.e. Figure 5C The last two columns of ) are stored in the floating diffusion node FD to complete the addition of pixel data of the same pixel row in the simulation stage.

[0063] Please refer to Figure 5A As shown, the reading circuit 23 reads the data sum of the pixel data of the first row of pixels (Row1) and the second row of pixels (Row2) at the same time after the exposure period of the second row of pixels ends; the data sum of the pixel data of the third row of pixels (Row3) and the fourth row of pixels (Row4) is read at the same time after the exposure period of the fourth row of pixels ends; and so on. In other words, the reading circuit 23 does not read the data of the floating diffusion node FD between the exposure periods of the Nth row of pixels and the N+1th row of pixels. For example, Figure 5AThere is no readout period after the exposure period of Row1 and Row3. That is, the data stored in FD and are read by turning on T4, which is Figure 5A Row2 and Row4 are turned on during the readout period after the exposure period.

[0064] Finally, the processor 25 generates an output image according to the plurality of data of the plurality of pixel rows. As described in the second embodiment above, the vertical resolution of the output image of the pixel array 21 is half the number of the plurality of pixel rows of the pixel array 21.

[0065] Similarly, the exposure period of the plurality of pixel rows of the pixel array 21 is less than or equal to 1 / 2 of the flashing period (for example, 1 / 120 seconds, but not limited to). Figure 5A The exposure period is shown as 1 / 240 sec.

[0066] Similarly, the exposure start time of the pixels in the N+1th row of the multiple pixel rows of the pixel array 21 is later than the exposure start time of the next pixel row. For example, the third exposure start time t3 of the pixels in the third row of the multiple pixel rows of the pixel array 21 is later than the first exposure start time t1 of the pixels in the first row of the multiple pixel rows and earlier than the second exposure start time of the pixels in the second row of the multiple pixel rows.

[0067] It must be noted that, although the above second and third embodiments are described by taking the addition of two pixel data of two adjacent pixel rows as an example, the present invention is not limited to this. In other implementations, image flicker can be eliminated by adding pixel data of more than two pixel rows. For example, when the exposure period of a pixel row is configured to be 1 / 3 of the ambient light flicker cycle, flicker interference can be reduced by adding (for example, in a digital stage or an analog stage) pixel data of three adjacent pixel rows; and so on. In similar Figure 5B In the implementation of adding pixel data in the analog phase, the photodiodes of three adjacent pixel rows may be connected to the same floating diffusion node to store the sum of the data.

[0068] In summary, conventional motion sensors may be affected by ambient light flicker and may make erroneous motion judgments. Therefore, the present invention further provides an operating method of a light sensor (see FIG. 3A to FIG. 5A ) and its pixel circuit (refer to Figure 5B ), which can at least reduce or even eliminate the flicker caused by time-varying ambient light by adding pixel data of two image frames or two adjacent pixel rows of one image frame without adjusting the image acquisition synchronization with the flicker frequency.

[0069] Although the present invention has been disclosed through the above examples, they are not intended to limit the present invention. Any person skilled in the art with ordinary knowledge in the art to which the present invention belongs can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the appended claims.

Claims

1. A light sensor for acquiring image frames relative to a flickering period of ambient light, comprising: A pixel array, the pixel array comprising a plurality of pixel rows, for acquiring pixel data with a rolling shutter; A reading circuit, the reading circuit is used to read the pixel data of the plurality of pixel rows respectively; as well as A processor is used to add the pixel data of the Nth row of pixels and the N+1th row of pixels of the multiple pixel rows respectively to generate a first output image, wherein N is a positive odd number starting from 1. 2 . The light sensor according to claim 1 , wherein a longitudinal resolution of the first output image is half the number of the plurality of pixel rows.

3. The optical sensor according to claim 1, in, The processor generates the first output image in the power saving mode, and the processor is further used in the normal mode to The pixel data of the Mth row of pixels and the M+1th row of pixels of the plurality of pixel rows are sequentially added to generate a second output image, wherein M is a positive integer starting from 1. 4 . The light sensor of claim 3 , wherein a longitudinal resolution of the second output image is smaller than the number of the plurality of pixel rows but greater than a longitudinal resolution of the first output image.

5. The light sensor according to claim 1, wherein a third exposure start time of pixels in a third row of the plurality of pixel rows is later than a first exposure start time of pixels in a first row of the plurality of pixel rows and earlier than a second exposure start time of pixels in a second row of the plurality of pixel rows. The light sensor according to claim 1 , wherein an exposure period of the plurality of pixel rows is ½ or ¼ of the blinking period.

7. The light sensor according to claim 1, wherein a time difference between an exposure start time of the N+1th row of pixels of the plurality of pixel rows and an exposure start time of the Nth row of pixels of the plurality of pixel rows is between 1 / 2 and 3 / 4 of the flashing period. 8 . The light sensor according to claim 1 , wherein an exposure period of the plurality of pixel rows is less than or equal to a half of the blinking period.

9. A light sensor for acquiring image frames relative to a flickering period of ambient light, comprising: A pixel array, the pixel array comprising a plurality of pixel rows, for acquiring pixel data with a rolling shutter; A reading circuit, the reading circuit is used to read the sum of the pixel data of the Nth row of pixels and the N+1th row of pixels of the plurality of pixel rows, respectively, wherein: N is a positive odd number starting from 1; as well as A processor is configured to generate an output image based on the plurality of data, wherein a vertical resolution of the output image is half the number of the plurality of pixel rows.

10. The light sensor according to claim 9, wherein the photodiode of each of the N-th row of pixels of the plurality of pixel rows and the photodiode of the adjacent pixel of the N+1-th row of pixels are connected to the same floating diffusion node for storing the data and. 11 . The light sensor according to claim 10 , wherein the read circuit does not read data of the floating diffusion node between exposure periods of the Nth row of pixels and the N+1th row of pixels. 12 . The light sensor according to claim 9 , wherein each of the pixels in the Nth row of the plurality of pixels is connected to the same readout line as an adjacent pixel in the N+1th row of pixels.

13. The light sensor according to claim 9, wherein a third exposure start time of pixels in a third row of the plurality of pixel rows is later than a first exposure start time of pixels in a first row of the plurality of pixel rows and earlier than a second exposure start time of pixels in a second row of the plurality of pixel rows. 14 . The light sensor according to claim 9 , wherein an exposure period of the plurality of pixel rows is less than or equal to a half of the blinking period.

Images