Mitigating flicker using event-based camera
By adjusting pixel parameters in an event-based camera to extend exposure time or change digital accumulation, the impact of flicker light sources on image capture is solved, improving safety in autonomous driving environments.
Patent Information
- Application Number
- CN202380062286.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2022-09-21
- Filing Date
- 2023-09-19
- Publication Date
- 2025-04-18
AI Technical Summary
Event-based cameras are susceptible to flickering light sources when capturing scenes of interest, resulting in incorrect image capture and autonomous decision-making, potentially endangering the lives of vehicles and pedestrians.
By operating a set of pixel parameters in an event-based camera, the values of different exposure time parameters and digital accumulation parameters are assigned to mitigate the impact of the flicker light source on the pixel. The specific steps include determining the area affected by the flicker light source, adjusting the pixel parameters to extend the exposure time or changing the digital accumulation, thereby reducing the pixel's sensitivity to the flicker light source.
It effectively reduces the impact of the flicker light source on event-based cameras, reduces the possibility of wrong image capture and autonomous decision-making, and improves the safety of the vehicle in autonomous driving environments.
Smart Images
Figure CN119856503A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for mitigating flicker light in a scene of interest imaged by an event-based camera, an assembly including an event-based camera and a computing device, and a vehicle including the assembly. Background Art
[0002] Light sources in cities, such as traffic lights or rear / front lights of cars, increasingly use LEDs (light emitting diodes) controlled by PWM (pulse width modulation) signals to control their brightness. However, the PWM-induced cycles of on and off brightness may be erroneously observed by conventional frame-based cameras, which capture images at equidistant points in time. Depending on the exposure time and frequency of such conventional frame-based cameras, the PWM-modulated brightness signal may be interpreted correctly or incorrectly as fully on, fully off, or as flickering at different frequencies, which may correspond to the PWM-signal frequency, or which may be different from the PWM-signal frequency. The situation is further complicated when light sources are used to transmit information while their brightness is modulated with a PWM signal, as in the case of, for example, car flashes.
[0003] In the case of autonomous driving, algorithmic processing of images of the environment that the vehicle is driving through is required for autonomous decision making. False signal detection as previously described may lead to erroneous autonomous decision making in such situations, thereby potentially endangering the lives of passengers in the autonomous vehicle and nearby pedestrians.
[0004] It is an object of the present invention to provide a method and arrangement for capturing an image of a scene of interest, wherein errors due to flicker are at least partially mitigated. Summary of the invention
[0005] According to a first aspect of the invention, there is provided a method for mitigating flicker light in a scene of interest imaged by an event-based camera, wherein the event-based camera comprises a plurality of pixels, wherein the imaging behavior of the plurality of pixels is affected by a set of pixel parameters comprising at least one exposure time parameter and / or at least one digital accumulation parameter, and wherein the method comprises the following steps:
[0006] 1) operating an event-based camera with assignment of a first value to a set of pixel parameters, the operation providing a plurality of events generated by the event-based camera;
[0007] 2) reconstructing an image based at least on the generated plurality of events,
[0008] 3) determining at least one region in the reconstructed image, the determined at least one region corresponding to at least one spatial region in a scene of interest, the scene of interest comprising at least one flickering light source active during operation of the event-based camera,
[0009] 4) determining a subset of pixel parameters in the set of pixel parameters that affect imaging behavior of pixels in the determined at least one region, assigning a second value to the determined subset of pixel parameters, determining or selecting the second value to mitigate flicker in pixels of the determined at least one region, and updating a first value assigned to the subset of pixel parameters with the second value, and 5) further operating the event-based camera with the updated set of pixel parameters.
[0010] In short and with necessary simplification, by assigning a second value to a subset of pixel parameters, the imaging behavior of pixels in at least one region can be changed. The exposure time can be extended by assigning a different value to at least one exposure time parameter, or the digital accumulation can be changed by assigning a different value to at least one digital accumulation parameter, which in turn can further stretch the exposure duration at the digital level. It may also be possible to assign different values to both. Extending and / or stretching the exposure time or duration can reduce the sensitivity of the pixels to flickering light sources, because under the assumption that the event-based camera is static, the flicker will no longer be perceived as a tail of continuous light pulses, and fewer events or no events are generated at all.
[0011] Light sources (such as light emitting diodes) controlled using pulse width modulation in traffic lights or in vehicle lights may be referred to as flashing light sources. Such light sources are typically present in scenes of interest that include road / traffic infrastructure. However, the present invention may also be used in other types of scenes of interest that include flashing light sources.
[0012] An event-based camera for capturing an image / imaging a scene of interest comprises a plurality of pixels (e.g. arranged as a pixel array), and the functionality / imaging behavior of the pixels is influenced / controlled by a set of pixel parameters (possible pixel parameters are, for example, exposure time(s) that influence the accumulation of an analog domain detection signal): each pixel typically comprises a photodetector (e.g. a photodiode) that converts light into, for example, an electric charge, and a detection signal (e.g. embodied as an electric charge) provided by the photodetector may be accumulated in the analog domain over a period of time (exposure time) (analog domain detection signal accumulation) before the accumulated analog domain detection signal is converted into a digital representation (sampled detection signal) using an analog-to-digital converter. The analog-to-digital converter is configured to sample the detection signal accumulated in the analog domain during the exposure time: thus, one exposure provides one sampled detection signal. The sampled detection signal (i.e. the analog domain detection signal accumulated during one exposure time) may be linearly dependent on the incident light intensity on the photodiode. Another possible pixel parameter may be a digital accumulation parameter: the sampled detection signal corresponding to the analog domain detection signal accumulated in the analog domain and sampled by the analog-to-digital converter may be (digitally) continuously accumulated (continuous accumulation may refer to the situation where the digital continuous accumulation does not introduce gaps relative to the underlying analog domain exposure; in other words, the exposure times of those sampled detection signals corresponding to the continuous accumulation may be aligned in time in such a way as to form a large joint exposure time substantially without gaps) a number of times equal to the integer value of the digital accumulation parameter; whereby the digital accumulation refers, for example, to a summation operation, the result of which may be referred to as a digitally accumulated sampled detection signal. Since summation is a linear operation, the digitally accumulated sampled detection signal is generally also linearly dependent on the incident light intensity.
[0013] An event-based camera may capture a scene of interest for a period of time, and during that period of time, the value assignments to a set of pixel parameters (or only a subset of the set of pixel parameters, where the term subset may refer to a proper subset, i.e., a subset may have a smaller cardinality than the set of pixel parameters) may be modified. In the case where the event-based camera comprises a pixel array having a plurality of rows and a plurality of columns, the sampled detection signals of the pixels of the pixel array may be read out row by row; thus, before revisiting a pixel, all other pixels of the pixel array may be visited first; an interval may be defined as a minimum time step, so that the exposure time, which may vary from pixel to pixel, always corresponds to an integer number of such intervals, and also always starts and ends at the boundaries of such intervals. Thus, at each boundary of such an interval, some pixels may be completing their exposure, and some of these pixels that have completed their exposure may generate corresponding events, as long as their event conditions are met. The readout speed of the pixels of the event-based camera may be adapted based on the length of the interval and / or optionally based on the number of columns and rows in the pixel array. An image may be reconstructed from the events generated based on the readout and processed sampled detection signals. Individual pixels or groups of pixels in the plurality of pixels may be associated with different pixel parameters. At a certain time, the event-based camera operates by assigning a first value to a set of pixel parameters, where the first value may be available a priori, such as embodied as an initialization parameter, or where the first value may have been determined or assigned based on previous operation of the event-based camera. In the course of the present disclosure, determination may be a synonym for assignment. They may be used interchangeably when technically feasible or useful.
[0014] Each of the generated multiple events may include i) information about a pixel or group of pixels among the multiple pixels with which it can be associated, ii) a timestamp at which it was generated, and / or iii) the type of event condition involved in its generation.
[0015] The different types will be explained in more detail below. Each event can be understood as a data entry or event data entry in a data object. Data objects such as these can be provided in the form of files on an electronic device external to the event-based camera or in the memory of the event-based camera itself.
[0016] The event condition can generally be based on a comparison of the digitally accumulated sampled detection signal of a pixel with a reference. For example, the event condition can be based on temporal contrast / difference, where the reference can be the previously digitally accumulated sampled detection signal of the same pixel; or the event condition can be based on spatial contrast / difference, where the reference can be the (current) digitally accumulated sampled detection signal of another pixel in the vicinity of the pixel; or the event condition can be based on spatial and temporal contrast / difference, where the reference can be the previously digitally accumulated sampled detection signal of another pixel in the vicinity of the pixel; or finally, the event condition can be based on simple thresholding, where the reference can be a fixed value. In this context, the previously digitally accumulated sampled detection signal may have been acquired as an integer multiple of the interval before the (current) digitally accumulated sampled detection signal of the pixel.
[0017] When the event condition is based on temporal contrast / difference and / or spatial contrast / difference, an event may be generated if the digitally accumulated sampled detection signal of a pixel differs from a reference by at least a threshold amount. The threshold amount may be a static value, or a dynamic value that depends on the digitally accumulated sampled detection signal and / or the reference, where the dependence may be linear, i.e. the threshold amount may be proportional to the digitally accumulated sampled detection signal and / or the reference, or the dependence may be non-linear, e.g. the threshold amount may be proportional to the square root of the digitally accumulated sampled detection signal and / or the reference. An event may only implicitly include the type of event condition involved in its generation, in that the event condition may not need to be explicitly included by the event, as long as it is known which event condition was involved in the generation of the event.
[0018] The correspondingly generated events may include polarity information, i.e., whether the difference is positive or negative, e.g., address information at which the event was generated, and time information at which the event was generated. Thus, each pixel of an event-based camera transmits information only when a quantity associated with the event condition has changed sufficiently, thereby potentially sparsely populating the output of the event-based camera. Prior to event generation, the digitally accumulated sampled detection signals may be pre-processed (in the digital domain), e.g., to remove inter-pixel mismatches caused by manufacturing process variations, in particular gain mismatches.
[0019] Reconstruction of an image can be performed in an incremental manner based on such events. For each pixel of the reconstructed image, where the reconstructed image can have the same size as the pixel array of the event-based camera, an update of the pixel value of the reconstructed image can occur when a new event is generated by the corresponding pixel of the event-based camera: for example, (one or more) threshold amounts (or multiples thereof) associated with the mentioned event conditions can be added / subtracted from the current pixel value of the reconstructed image to obtain an updated reconstructed image. Other types of reconstruction of images from events are known from the prior art; typically, the reconstruction depends on the type of event conditions used during the generation of the event, in particular on the type of information contained in the event, i.e., whether the event includes information about the sign and / or about the amplitude of the temporal contrast / difference and / or about the spatial contrast / difference. An image can also be reconstructed by processing groups of events together and by updating the reconstructed image based on the processed groups of events. Event-based Vision: A Survey by Gallego et al.; IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020; https: / / doi.org / 10.1109 / TPAMI.2020.3008413 Examples of methods for reconstructing images from events are disclosed in They can also be used to reconstruct images from events even if the event-based camera captures flickering light sources.
[0020] Then, in the reconstructed image, (one or more) regions are subsequently determined, which can be associated with a portion of a spatial scene of interest in which a flickering light source may be present. Subsequently, a subset of pixel parameters in a set of pixel parameters is determined, which is associated with the determined (one or more) regions, where association means that changing the pixel parameters in the subset of pixel parameters will change the imaging behavior of the pixels in the determined (one or more) regions. A suitable second value is determined or assigned for the subset of parameters, which mitigates flicker in the determined (one or more) regions in the case of further operation of the event-based camera. The second value of the pixel parameter associated with the pixel in the determined (one or more) regions may vary from pixel to pixel in the determined (one or more) regions. For example, pixel parameters that affect the imaging behavior of pixels that are not in at least one of the determined regions can be set to optimize the signal-to-noise ratio (SNR) and / or temporal resolution. In this way, the imaging behavior of only those pixels associated with flicker is suitable for mitigating flicker, while the remaining pixels can be suitable for optimizing the overall imaging behavior based on specific requirements that may be attempted to be achieved (e.g., attempting to achieve a minimum signal-to-noise ratio).
[0021] In an embodiment of the method according to the present invention, the event conditions involved in the generation of the multiple events are embodied as time contrast conditions and / or as time difference conditions and / or as space contrast conditions and / or as space difference conditions.
[0022] In a further embodiment of the method according to the present invention, wherein the analog domain detection signal accumulation of each pixel is affected by the at least one exposure time parameter (using the exposure time parameter associated with the pixel, the duration of the exposure time can be controlled and set), the detection signal is provided by a photodetector included by the corresponding pixel, wherein each pixel or pixel group of the plurality of pixels is associated with an exposure time parameter of the at least one exposure time parameter.
[0023] In a further embodiment of the method according to the present invention, the plurality of pixels are arranged as a pixel array having a plurality of rows and a plurality of columns, wherein the event-based camera comprises a readout processor, and wherein the readout processor is configured to continuously read out a row from the plurality of rows one after another during operation of the event-based camera and generate the plurality of events.
[0024] In a further embodiment of the method according to the invention, in case the generated multiple events are generated based on a temporal contrast condition and / or based on a temporal difference condition, an image is reconstructed based on i) the multiple events and optionally ii) at least one previously generated image. If the events contain intensity information, the image can be reconstructed based on the events only. If the events contain contrast / difference information, the image can be reconstructed based on the events and the previously generated images.
[0025] In a further embodiment of the method according to the present invention, determining at least one region in the reconstructed image comprises the following steps: a) determining at least one histogram of the reconstructed image, b) performing threshold processing on the reconstructed image based on the determined at least one histogram, wherein the threshold processing provides two thresholded reconstructed images, c) filtering out noisy bright pixels in the two thresholded reconstructed images using at least one morphological function, and d) using a spot detection technique, determining bright spots in the two thresholded reconstructed images, and selecting from the determined bright spots those bright spots whose shapes are substantially similar in the two thresholded reconstructed images, wherein the selected bright spots are the determined at least one region.
[0026] One or more histograms may be determined from the reconstructed image. For example, if more than one histogram is determined, different bin sizes may be used for different histograms. For thresholding of the reconstructed image, one or two thresholds may be used: (i) for example, in the case of one threshold and two histograms, two thresholded reconstructed images may be determined by thresholding the two histograms with one threshold, respectively; (ii) for example, in the case of two thresholds (which may be different from each other) and one histogram, two thresholded reconstructed images may be determined by thresholding one histogram with two thresholds, respectively. The threshold(s) used for thresholding may be determined based on the histogram(s), or alternatively provided a priori. The morphological function may be, for example, erosion, dilation, opening (first erosion, then dilation in series), closing (first dilation, then erosion in series), proper-opening, proper-closing, or automatic median. Such morphological functions may be appropriately combined to filter out noisy bright or dark pixels from the two thresholded reconstructed images. Bright spots may be determined in two thresholded reconstructed images using known algorithms for blob detection; such bright spots may be compared across the two thresholded reconstructed images, and those bright spots may be used as the determined at least one region that is substantially found in both thresholded reconstructed images. In an alternative embodiment, only one thresholded reconstructed image is determined and compared to the reconstructed image to determine suitable bright spots. In addition to using histogram(s) and threshold(s), the at least one region may alternatively be determined as follows: gamma correction with two different power parameters γ may be employed to provide two different gamma-corrected reconstructed images; the two gamma-corrected reconstructed images may then be processed as described for the two thresholded reconstructed images.
[0027] In a further embodiment of the method according to the invention, determining at least one region in the reconstructed image comprises applying a trained neural network to the reconstructed image, wherein the trained neural network provides the at least one region as output in the form of at least one bounding box and / or in the form of at least one anchor box and / or in the form of a segmentation mask and / or in the form of a probability heat map. The reconstructed image that can be provided as input to the neural network can be embodied as a grayscale image or a color image of any bit depth for each of the image channels. Optionally, the neural network can also take the above-mentioned histogram as input. The neural network can include multiple layers, some of which are potentially embodied as convolutional layers. The neural network can also include one, two or more nonlinear activation functions and / or normalization layers. Some of the layers of the neural network may have jump connections between them. The neural network can be trained using a collection of annotated reconstructed images and / or their corresponding histograms, where the bright spots are annotated by humans in the form of bounding boxes. The annotations can all contain the same label, such as "light", or can distinguish different types of bright spots, such as "car lights", "street lights", etc. Before training, you can initialize the neural network with random numbers and train the neural network from scratch, or you can pre-train some of the layers using publicly available classification datasets.
[0028] In a further embodiment of the method according to the invention, the determined subset of pixel parameters comprises at least one exposure time parameter and at least one digital accumulation parameter, and wherein assigning a second value to at least one exposure time parameter and at least one digital accumulation parameter comprises the following steps: a) assigning a required total accumulation duration, the required total accumulation duration being based on a flickering frequency of at least one flickering light source active during operation of the event-based camera, or receiving a desired total accumulation duration, b) determining a second value for at least one exposure time parameter based on the determined required total accumulation duration or the desired total accumulation duration and sampled detection signals of pixels in at least one determined area obtained during operation of the event-based camera in such a way that the pixels in at least one determined area are not oversaturated during further operation, and c) assigning a second value to at least one digital accumulation parameter based on the second value determined for at least one exposure time parameter and the determined required total accumulation duration or the desired total accumulation duration.
[0029] The first value of the exposure time parameter may have been determined to maximize the signal-to-noise ratio (SNR) of the pixel and prevent saturation of the pixel at the same time. One of the main sources of noise in image sensors is photon shot noise; in that case, the SNR is approximately equal to the square root of the total number of photons collected during one exposure. Therefore, as long as the pixel is not saturated, it may be desirable to have a longer exposure time. However, the longer the exposure time, the lower the sampling rate (i.e., the temporal resolution) of the pixel. Therefore, in general, there is a trade-off between the SNR and the temporal resolution of the pixel, and the optimal point may generally depend on the application. For example, for automotive applications, it may be necessary to ensure that the temporal resolution is at least 30Hz, which means that even in this case the SNR is compromised, the exposure time cannot be set to more than 33ms. On the other hand, optimizing the temporal resolution may mean setting the exposure time as short as possible given the SNR required based on the application. Since the SNR is approximately the square root of the cumulative signal, the cumulative signal can be evaluated to estimate the SNR. For example, if the application requires an SNR of "100", which corresponds to a cumulative signal of "500", then the exposure time can be set to the shortest value, which results in a cumulative signal of at least "500".
[0030] The first value of the digital accumulation parameter may be set to "1", i.e., the digital accumulation contains only one sampled detection signal / sums one sampled detection signal to maximize the temporal resolution of the pixel. Alternatively, the first value of the digital accumulation parameter may be set to achieve a certain desired total accumulation duration corresponding to the desired temporal resolution of the pixel based on the application, e.g., a desired temporal resolution of 30 Hz corresponds to a desired total accumulation duration of about 33 ms. In particular, the first value of the digital accumulation parameter may be derived as the desired total accumulation duration divided by the first value of the exposure time parameter.
[0031] Flicker mitigation can be performed directly by changing (one or more) exposure time parameters. As long as the pixel is not saturated, the second value of the exposure time parameter can be set to be equal to or greater than the desired total accumulation duration. In order to check or predict (project) whether the pixel is saturated or may become saturated, the sampled detection signal of the pixel can be compared with a predetermined potential saturation threshold. If the sampled detection signal corresponds to, for example, a 10-bit value, the potential saturation threshold can be set to a value less than the maximum 10-bit value, such as "500" or "700". For example, in the case where the desired total accumulation duration is 16ms, if the second value of the exposure time parameter can be set to 16ms without saturating the pixel, the sampled detection signal obtained in the previous 16ms can be evaluated for prediction. Alternatively, the second value of the exposure time parameter can be directly set to 16ms, and if the subsequent sampled detection signal shows that the pixel is saturated, the second value of the exposure time parameter can be readjusted (for example, in the case where the sampled detection signal exceeds the potential saturation threshold, the second value of the exposure time parameter can be reduced, for example, by 50%).
[0032] If the second value of the exposure time parameter is set to a desired total accumulation duration that will saturate the pixels, then in this case, flicker mitigation will be achieved with the aid of (one or more) digital accumulation parameters. In this case, the second value of the exposure time parameter can be set equal to the first value, or higher than the first value, i.e., the highest value without saturating the pixels. For example, in the case where the second value of the exposure time parameter is set to 8ms to prevent saturating the pixels, and the desired total accumulation duration is 16ms, the second value of the digital accumulation parameter will be set to "2".
[0033] Typically, the second value of the digital accumulation parameter can be derived as the desired total accumulation duration divided by the second value of the exposure time parameter. In the case where there is also a desired total accumulation duration set by the application, the second value of the digital accumulation parameter can be derived by dividing the higher value between the desired total accumulation duration and the desired total accumulation duration by the second value of the exposure time parameter.
[0034] Preferably, both the exposure time parameter and the required total accumulation duration can be set to a power of "2" of a previously defined interval (corresponding to the minimum exposure time): setting the exposure time and the required total accumulation duration in this way can simplify the derivation of the digital accumulation parameters and can simplify the subsequent normalization of the digitally accumulated sampled detection signal.
[0035] In general, assuming that the first values of the exposure time parameter and the digital accumulation parameter of the pixel have been determined to optimize the SNR and / or the temporal resolution of the pixel based on the application, the following steps can be performed to determine the second values of the exposure time parameter and the digital accumulation parameter of the pixels within the determined at least one region: first, check whether the first value of the exposure time parameter is longer than the total required accumulation duration - if so, the second values of the exposure time parameter and the digital accumulation parameter can be set equal to their corresponding first values; if not, check whether the second value of the exposure time parameter can be set to the total required accumulation duration without saturating the pixel - if so, set the second value of the exposure time parameter to the total required accumulation duration; if not, set the second value of the exposure time parameter to at least the corresponding first value, or higher than the corresponding first value, without saturating the pixel. Secondly, after determining the second value of the exposure time parameter, determine the second value of the digital accumulation parameter based on the second value of the exposure time parameter, the total required accumulation duration or the expected accumulation duration. These steps can preferably be performed by the readout processor.
[0036] Thus, the second values of the exposure time parameter and the digital accumulation parameter may differ between pixels. The pixels within the determined at least one region may have had their pixel parameters set to achieve flicker mitigation as a first priority goal, and to optimize SNR and / or temporal resolution based on the application as a secondary goal. On the other hand, the pixels not within the determined at least one region may have had their pixel parameters set to optimize SNR and / or temporal resolution based on the application as the only goal.
[0037] In a further embodiment of the method according to the invention, the flicker frequency is assumed to be greater than 50 Hz, in particular greater than 80 Hz, and wherein the required total accumulation duration is determined in such a way that in a reconstructed image based on a plurality of events generated during further operation of the event-based camera, flicker is mitigated. Alternatively, the flicker frequency is assumed to be greater than 50 Hz, in particular greater than 80 Hz, and wherein the required total accumulation duration is selected to be greater than the reciprocal of the assumed flicker frequency, such that in a reconstructed image based on a plurality of events generated during further operation of the event-based camera (6), flicker is mitigated. The reciprocal of the assumed flicker frequency may correspond to a flicker period of the flickering light source.
[0038] Since the exposure time parameters and the digital accumulation parameters can be determined and are therefore known, the digitally accumulated sampled detection signal can be normalized based on this knowledge, for example, the digitally accumulated sampled signal can be normalized by dividing by the total number of intervals corresponding to the digital accumulation. Many flickering light sources in a scene of interest (such as a road network) can be assumed to have a known flicker frequency. For example, in the case where the method according to the present invention is used in such a scene of interest, the flicker frequency may be known and therefore does not need to be determined. For example, 50 Hz and 80 Hz may correspond to the lowest flicker encountered in an automotive use case; these frequencies or higher frequencies are also generally not perceptible to the human eye and therefore need to be mitigated by an event-based camera in order to be correctly interpreted.
[0039] In a further embodiment of the method according to the invention, each pixel of the plurality of pixels comprises its own exposure time parameter, or wherein groups of pixels of the plurality of pixels share exposure time parameters.
[0040] Each pixel in the plurality of pixels may also include its own digital accumulation parameter, or groups of pixels in the plurality of pixels may share digital accumulation parameters.
[0041] In a further embodiment of the method according to the invention, the at least one flashing light source is embodied as a traffic light source and / or as a vehicle light source.
[0042] In a further embodiment of the method according to the invention, the plurality of events are generated at multiples of a period of a global clock of the event-based camera. Specifically, the plurality of events may be generated at the boundaries of a previously defined interval, wherein the exposure time may be an integer multiple of the interval.
[0043] According to a second aspect of the present invention, a component is provided, comprising: a) an event-based camera, comprising at least: i) a plurality of pixels arranged as a pixel array, ii) a readout processor, and iii) a parameter memory for storing a set of pixel parameters, and b) a computing device, in particular embodied as a microprocessor, wherein the event-based camera is configured to transmit generated events to the computing device, wherein the component is configured to perform the method according to the first aspect of the present invention, wherein the event-based camera is configured to perform method steps 1), 4) and 5) of the method according to the first aspect of the present invention, and wherein the computing device is configured to perform method steps 2) and 3) of the method according to the first aspect of the present invention.
[0044] In an embodiment of the assembly according to the second aspect of the invention, the computing device is part of the event-based camera, or the computing device is an external computing device separate from the event-based camera.
[0045] According to a third aspect of the invention, there is provided a vehicle comprising an assembly according to the second aspect of the invention, wherein the vehicle is configured to be driven in a scene of interest comprising traffic lights and / or lights of other vehicles.
[0046] For example, the vehicle may be embodied as a conventional automobile or a motorcycle. BRIEF DESCRIPTION OF THE DRAWINGS
[0047] Exemplary embodiments of the present invention are disclosed in the specification and illustrated by the accompanying drawings, in which:
[0048] Figure 1 schematically illustrates an embodiment of a method for mitigating flicker light in a scene of interest imaged by an event-based camera;
[0049] Figure 2 schematically illustrates an event-based camera imaging a scene of interest, the scene including a traffic light source as a flashing light source;
[0050] Figure 3 schematically illustrates an event-based camera imaging a scene of interest, the scene including a car including at least one headlight as a flashing light source; and
[0051] Figure 4 An embodiment of an assembly according to the invention is schematically illustrated. DETAILED DESCRIPTION
[0052] Figure 1 An embodiment of a method for mitigating flicker light in a scene of interest imaged by an event-based camera is schematically illustrated. The event-based camera is operated so as to image a scene of interest, the imaging providing a plurality of events 2 as output. The event-based camera comprises a plurality of pixels, the imaging behavior of the pixels being affected by a set of pixel parameters. The event-based camera is operated with a first value assigned to the set of pixels. Based at least on the plurality of events 2, an image is reconstructed 3, and in the reconstructed image, at least one region is determined 4, the region corresponding to a spatial region in the scene of interest potentially comprising a flicker light source. Subsequently, a subset of pixel parameters in the set of pixel parameters is determined 5, wherein the subset of pixel parameters is associated with pixels in the determined at least one region, and for the subset of pixel parameters, a second value is determined 5, the second value being determined to mitigate flicker in corresponding pixels of the determined at least one region. The first value of the subset of pixel parameters is updated / replaced 5 with the second value determined 5 for the subset of pixel parameters before further operating 6 the event-based camera with the updated pixel parameters. Thus, flicker in the determined at least one region is mitigated in the reconstructed image(s) based on the further operation 6.
[0053] Figure 2 An event-based camera is schematically illustrated for imaging a scene of interest, the scene including traffic lights as flickering light sources. The event-based camera 7 may be installed / placed, for example, in a car configured for autonomous driving; thus, the event-based camera 7 may provide information to a driving algorithm steering the car by imaging the scene of interest through which the car drives. Traffic lights 8 (including traffic lights 8 of light emitting diodes) in the scene of interest may be controlled by a pulse width modulated signal, which in turn causes flicker. Using the method according to the invention, such flicker is mitigated by a suitable combination of algorithmic processing and hardware settings of the event-based camera 7.
[0054] Figure 3 An event-based camera is schematically illustrated imaging a scene of interest, the scene comprising a car including at least one headlight as a flashing light source. Figure 3 Similar in structure to Figure 2 :exist Figure 3 In the embodiment, the flashing light source is embodied as a headlight 9 of another vehicle, and the event-based camera 7 can be as follows Figure 2 The invention is mounted on a vehicle driving through a scene of interest, in which another vehicle, such as a car including at least one headlight, is also driving.
[0055] Figure 4 An embodiment of a component according to the present invention is schematically illustrated. The component comprises an event-based camera 7 and a computing device 10. The computing device 10 is configured to reconstruct an image from a plurality of events generated by imaging a scene of interest by the event-based camera 7, and to determine at least one region in the reconstructed image. The computing device 10 receives a plurality of events from the event-based camera 7 as input, and provides coordinates of pixels in the determined at least one region as output to the event-based camera 7, and then the event-based camera 7 (specifically, a readout processor of the event-based camera) determines a second value of a subset of pixel parameters associated with the pixels in the determined at least one region; the event-based camera 7 then updates its corresponding hardware parameters. Figure 4 In the embodiment, the computing device is physically separated from the event-based camera, for example located in the trunk of a car in which the assembly is placed. Alternatively, the computing device 10, typically embodied as a microprocessor, can be part of the event-based camera 7, which means that all processing can be done in the event-based camera 7.
Claims
1. A method for mitigating flicker light in a scene of interest imaged by an event-based camera (7), wherein the event-based camera (7) comprises a plurality of pixels, wherein the imaging behavior of the plurality of pixels is influenced by a set of pixel parameters comprising at least one exposure time parameter and / or at least one digital accumulation parameter, and wherein the method comprises the following steps: 1) operating (1) an event-based camera (7) having an assignment of a first value to a set of pixel parameters, said operation (1) providing a plurality of events (2) generated by the event-based camera (7); 2) reconstructing (3) an image based at least on the generated plurality of events (2), 3) determining (4) at least one region in the reconstructed image, the determined at least one region corresponding to at least one spatial region in a scene of interest, the scene of interest comprising at least one flickering light source (8, 9) active during operation of the event-based camera (7), 4) determining (5) a subset of pixel parameters from the set of pixel parameters, the subset of pixel parameters affecting the imaging behavior of pixels in the determined at least one region; assigning (5) a second value to the determined subset of pixel parameters, the second value being determined to mitigate flicker in pixels of the determined at least one region; and updating (5) the first value assigned to the subset of pixel parameters with the second value; as well as 5) Further operate (6) the event-based camera (7) with the updated set of pixel parameters.
2. The method according to claim 1, wherein: The event conditions involved in the generation of the multiple events (2) are embodied as time contrast conditions and / or as time difference conditions and / or as space contrast conditions and / or as space difference conditions.
3. A method according to claim 1 or 2, wherein the analog domain detection signal accumulation of each pixel is affected by the at least one exposure time parameter, the detection signal is provided by a photodetector included in the corresponding pixel, and each pixel of the multiple pixels is associated with an exposure time parameter in the at least one exposure time parameter.
4. A method according to any one of the preceding claims, wherein the plurality of pixels are arranged in a pixel array having a plurality of rows and a plurality of columns, wherein the event-based camera (7) comprises a readout processor, and wherein the readout processor is configured to continuously read out a row from the plurality of rows one after another during operation of the event-based camera (7) and generate the plurality of events (2).
5. The method according to any one of claims 2 to 4, wherein: In case the generated plurality of events (2) are generated based on a temporal contrast condition and / or based on a temporal difference condition, an image is reconstructed (3) based on i) the plurality of events (2) and ii) at least one previously generated image.
6. A method according to any one of the preceding claims, wherein: Determining (4) at least one region in the reconstructed image includes the following steps: a) determining at least one histogram of the reconstructed image, b) thresholding the reconstructed image based on the determined at least one histogram, wherein the thresholding provides two thresholded reconstructed images, c) filtering out noisy bright pixels in the two thresholded reconstructed images using at least one morphological function, and d) using a spot detection technique, determining bright spots in the two thresholded reconstructed images, and selecting from the determined bright spots those whose shapes are substantially similar in the two thresholded reconstructed images, wherein the selected bright spots are the determined at least one region.
7. The method according to any one of claims 1 to 5, wherein: Determining (4) at least one region in the reconstructed image comprises applying a trained neural network to the reconstructed image, wherein the trained neural network provides the at least one region as an output in the form of at least one bounding box and / or in the form of at least one anchor box and / or in the form of a segmentation mask and / or in the form of a probability heat map.
8. The method according to any one of claims 3 to 7, wherein the determined subset of pixel parameters comprises at least one exposure time parameter and at least one digital accumulation parameter, and wherein assigning (5) a second value for at least one exposure time parameter and at least one digital accumulation parameter comprises the following steps: a) determining a required total accumulation duration, the required total accumulation duration being based on a flashing frequency of at least one flashing light source (8, 9) active during operation (1) of the event-based camera (7), or receiving a desired total accumulation duration, b) assigning (5) a second value to at least one exposure time parameter based on the determined required total accumulation duration or the desired total accumulation duration and sampled detection signals of pixels in at least one determined area obtained during operation (1) of the event-based camera (7) in such a way that the pixels in the determined at least one area are not oversaturated during further operation (6), and c) assigning (5) a second value to at least one digital accumulation parameter based on the second value determined for at least one exposure time parameter and the determined required total accumulation duration or the desired total accumulation duration.
9. A method according to claim 8, wherein a flicker frequency is assumed to be greater than 50 Hz, in particular greater than 80 Hz, and wherein the required total integration duration is selected to be greater than the inverse of the assumed flicker frequency, so that in a reconstructed image based on a plurality of events generated during further operation (6) of the event-based camera (7), flicker is mitigated.
10. The method of any one of claims 8 or 9, wherein each pixel of the plurality of pixels comprises its own exposure time parameter, or wherein groups of pixels of the plurality of pixels share exposure time parameters.
11. The method according to any of the preceding claims, wherein the at least one flashing light source (8, 9) is embodied as a traffic light source (8) and / or as a vehicle light source (9).
12. A method according to any one of the preceding claims, wherein: The plurality of events (2) are generated at multiples of a period of a global clock of the event-based camera (7).
13. An assembly (7, 10) comprising: a) an event-based camera (7) comprising at least: i) a plurality of pixels arranged as a pixel array, ii) a readout processor, and iii) a parameter memory for storing a set of pixel parameters, and b) a computing device (10), in particular embodied as a microprocessor, wherein the event-based camera (7) is configured to transmit generated events (2) to the computing device (10), wherein the components (7, 10) are configured to perform a method according to any one of claims 1 to 12, wherein the event-based camera (7) is configured to perform method steps 1), 4) and 5), and wherein the computing device (10) is configured to perform method steps 2) and 3).
14. The assembly (7, 10) of claim 13, wherein the computing device (10) is part of the event-based camera (7), or wherein the computing device (10) is an external computing device separate from the event-based camera (7).
15. A vehicle comprising an assembly according to any one of claims 13 or 14, wherein the vehicle is configured to drive in a scene of interest comprising traffic lights (8) and lights (9) of other vehicles.