Using an event-based camera to reduce flicker

Abstract

A method and assembly for capturing a scene of interest with reduced errors due to light flicker is provided. The present invention relates to a method for reducing light flicker in a scene of interest imaged by an event-based camera (7), the event-based camera (7) comprising a plurality of pixels, The imaging behavior of the plurality of pixels is influenced by a pixel parameter set comprising at least one exposure time parameter and / or at least one digital accumulation parameter, and the method comprises: 1) operating an event-based camera (7) with assigning a plurality of first values ​​to the pixel parameter sets, operating an event-based camera (7), the operation (1) providing a plurality of events (2) generated by the event-based camera (7); 2) reconstructing (3) an image based at least on the generated plurality of events (2); 3) determining (4) at least one region in the reconstructed image, wherein the determined at least one region corresponds to at least one spatial region in the scene of interest comprising at least one flickering light source (8, 9) active during the operation of the event-based camera (7); determining (4) at least one region within the reconstructed image; 4) determining a subset of pixel parameters (5) of the set of pixel parameters, the subset of pixel parameters affecting the imaging behavior of a plurality of pixels within the determined at least one region; assigning (5) second values ​​to the determined subset of pixel parameters, the second values ​​being determined to reduce flicker within pixels of the determined at least one region; and updating (5) the first values ​​assigned to the subset using the second values; and 5) further operating (6) the event-based camera (7) using the updated pixel parameter set; The invention also relates to an assembly and a vehicle comprising this assembly.

Description

[Technical Field]

[0001] The present invention relates to a method for reducing flickering light in a scene of interest imaged with an event-based camera, an assembly including an event-based camera and a computing device, and a vehicle including the assembly. [Background technology]

[0002] Light-emitting diodes (LEDs), whose brightness is controlled by a pulse-width modulation (PWM) signal, are increasingly being used in street traffic lights and automobile taillights and headlights. However, the PWM-controlled light-dark cycles can be misinterpreted by traditional frame-based cameras, which capture images at evenly spaced intervals. Depending on the exposure time and frequency of such cameras, the PWM-modulated brightness signal may be interpreted correctly or differently from the PWM signal frequency, such as fully lit (ON), fully extinguished (OFF), or flickering at different frequencies corresponding to the PWM signal frequency. This situation becomes even more complicated when the light source communicates information by modulating its brightness with a PWM signal, such as a car's turn signal.

[0003] In autonomous driving situations, algorithmic processing of images of the environment in which the vehicle is driving is necessary for autonomous decision-making. The aforementioned erroneous signal detection can lead to erroneous autonomous decision-making in such situations, thereby endangering the lives of passengers in the autonomous vehicle and nearby pedestrians. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Gallego et al.; Event-based Vision: A Survey; IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020; https: / / doi.org / 10.1109 / TPAMI.2020.3008413 Summary of the Invention [Problem to be solved by the invention]

[0005] It is an object of the present invention to provide a method and assembly for capturing an image of a scene of interest in which errors due to light flicker are at least partially mitigated. [Means for solving the problem]

[0006] In a first aspect of the present invention, there is provided a method for reducing light flicker in a scene of interest imaged by an event-based camera, the method comprising: the event-based camera comprises a plurality of pixels; The imaging behavior of the plurality of pixels is influenced by a pixel parameter set comprising at least one exposure time parameter and / or at least one digital accumulation parameter, and the method comprises: 1) operating an event-based camera with assigning a plurality of first values ​​to the pixel parameter sets; operating an event-based camera, the operation providing a plurality of events generated by the event-based camera; 2) reconstructing an image based at least on the generated events; 3) determining at least one region within the reconstructed image, the determined at least one region corresponding to at least one spatial region within the scene of interest comprising at least one flickering light source that is active during the operation of the event-based camera; determining at least one region within the reconstructed image; 4) determining a subset of pixel parameters of the set of pixel parameters, the subset of pixel parameters affecting the imaging behavior of a plurality of pixels within the determined at least one region; assigning second values ​​to the determined subset of pixel parameters, the second values ​​being determined to reduce flicker within pixels in the determined at least one region; and updating the first values ​​assigned to the subset of pixel parameters using the second values; and 5) further operating the event-based camera using the updated pixel parameter set; Equipped with.

[0007] In simple terms, and with necessary simplifications, the imaging behavior of pixels in at least one region can be changed by assigning second values ​​to a subset of pixel parameters. The exposure time can be extended by assigning a different value to at least one exposure time parameter, or the digital accumulation can be modified by assigning a different value to at least one digital accumulation parameter, in return for further extending the exposure time at the digital level. Also, different values ​​can be assigned to both. By slowing and / or extending the exposure time or exposure period, the sensitivity of the pixels to flickering light sources can be reduced, since, under the assumption that the event-based camera is static, the flicker is no longer perceived as a trail of successive light pulses, and fewer or no events occur.

[0008] Light sources such as light emitting diodes controlled using pulse width modulation in traffic signals or automobile lights may be referred to as flickering light sources. Such light sources are typically present in scenes of interest that include roads or transportation infrastructure. However, the present invention may also be employed in other types of scenes of interest that include flickering light sources.

[0009] An event-based camera that captures or images an image of a scene of interest comprises a plurality of pixels arranged, for example, as a pixel array, whose function or imaging behavior is influenced or controlled by a set of pixel parameters, possible pixel parameters being, for example, exposure time(s) that affect the accumulation of a detected signal in the analog domain. Each pixel typically comprises a photodetector (e.g., a photodiode) that converts light into, for example, an electric charge, and the detection signal provided by the photodetector, e.g., embodied as an electric charge, may be accumulated in the analog domain over a period of time called the exposure time (accumulation of the analog domain detection signal) before the accumulated analog domain detection signal is converted into a digital representation (sampled detection signal) using an analog-to-digital converter. The analog-to-digital converter is adapted to sample the detection signal accumulated in the analog domain during the exposure time. Thus, one exposure provides one sampled detection signal. The sampled detection signal (the detection signal in the analog domain accumulated during one exposure time) may be linearly dependent on the light intensity incident on the photodiode. Another possible pixel parameter may be a digital accumulation parameter. The sampled detection signals corresponding to the analog domain detection signals accumulated in the analog domain and sampled by the analog-to-digital converter may be accumulated (digitally) consecutively multiple times (consecutive accumulation may refer to a case where the digital consecutive accumulation does not produce gaps with respect to the underlying analog domain exposure. In other words, the exposure times corresponding to those sampled detection signals that are accumulated consecutively may be aligned in time to form one long, continuous exposure time with substantially no gaps). The "numerous times" is equal to the integer value of the digital accumulation parameter. Thus, digital accumulation refers to, for example, an integration operation, and the result of the digital accumulation may be referred to as a digitally accumulated sampled detection signal. Because integration is a linear operation, the digitally accumulated sampled detection signal typically also has a linear dependence on the incident light intensity.

[0010] An event-based camera may capture a scene of interest over a period of time, during which time the assignment of values ​​to a pixel parameter set (or to only a subset of the pixel parameter set, where the term subset may refer to an appropriate subset, i.e., the subset may have smaller cardinality than the pixel parameter set) may be changed. If the event-based camera comprises a pixel array with multiple rows and multiple columns, the sampled detection signals of the pixels of the pixel array may be read out row by row, and before a pixel is revisited, all other pixels of the pixel array may be visited first. An interval may be defined as the smallest time step such that the different exposure times for each pixel always correspond to an integer number of intervals and always start and end at the interval boundaries. Thus, at each such interval boundary, there is a pixel that completes exposure, and some of the completed exposure pixels may generate a respective event, provided that its event condition is met. The readout rate of the pixels of the event-based camera may be adapted based on the length of the interval or, alternatively, based on the number of columns and rows of the pixel array. An image may be reconstructed from the generated events based on the read-out and processed sampled detection signals. Individual pixels or groups of pixels of the plurality of pixels may be associated with different pixel parameters. At some point, the event-based camera is operated with assignment of first values ​​to a set of pixel parameters, where the first values ​​may be available a priori (e.g., may be embodied as initialization parameters, or may be determined or assigned based on previous operation of the event-based camera). Determining may be synonymous with assigning in the course of this disclosure. These terms (determining and allocating) may be used interchangeably where technically feasible or useful.

[0011] Each event in the generated events is i) information about a pixel or group of pixels of a plurality of pixels that can be associated; ii) the timestamp at which it was generated; and iii) the type of event conditions involved in its creation; It may include at least one of the following. The different types are explained in more detail below. Each event may be understood as a data entry or event data entry within a data object. The data object may be provided in the form of a file in an electronic device external to the event-based camera or in the memory of the event-based camera itself.

[0012] The event condition may generally be based on a comparison of a digitally accumulated sampled detection signal of a pixel with a reference. For example, the event condition may be based on temporal contrast or difference. The reference may be a previous digitally accumulated sampled detection signal of the same pixel, or the event condition may be based on spatial contrast or difference. The reference may be a (current) digitally accumulated sampled detection signal of another pixel in the pixel's neighborhood, or the event condition may be based on spatial and temporal contrast or difference. The reference may be a previous digitally accumulated sampled detection signal of another pixel in the pixel's neighborhood. Finally, the event condition may alternatively be based on a simple threshold, and the reference may be a fixed value. Here, the previous digitally accumulated sampled detection signal may be an integer multiple of the previous interval of the (current) digitally accumulated sampled detection signal of the pixel.

[0013] If the event condition is based on temporal contrast / difference and / or spatial contrast / difference, an event may be generated when the digitally stored sampled detection signal of a pixel differs from a reference by at least a threshold value. The threshold value may be a static value or may be a dynamic value that depends on the digitally stored sampled detection signal and / or the reference. The dependence may be linear, i.e., the threshold may be proportional to one or both of the digitally stored sampled detection signal and the reference, or the dependence may be non-linear, i.e., the threshold may be proportional to the square root of one or both of the digitally stored sampled detection signal and the reference. An event only implicitly carries the type of event condition that was involved in its creation, since this event condition does not need to be explicitly carried by the event, as long as it is known which event conditions were involved in the creation of the event.

[0014] The correspondingly generated event may comprise polarity information, whether the spread was positive or negative, e.g., address information at which the event was generated, and timing information at which the event was generated. Thus, each pixel of the event-based camera transmits information only if a quantity related to the event condition has changed sufficiently, which may result in a sparsifying output of the event-based camera. Prior to event generation, the digitally accumulated sampled detection signal (in the digital domain) may be pre-processed to remove pixel-to-pixel discrepancies, particularly gain discrepancies, due to, for example, manufacturing process variations.

[0015] Image reconstruction may be performed incrementally based on such events: for each pixel in the reconstructed image, the reconstructed image may have the same size as the pixel array of the event-based camera, and updating of the pixel value in the reconstructed image may occur when the corresponding pixel in the event-based camera generates a new event. For example, a threshold value (or a multiple thereof) associated with the mentioned event condition may be added or subtracted from the current pixel value of the reconstructed image to obtain an updated reconstructed image. Other types of image reconstruction from events are known from the prior art: in general, the reconstruction depends on the type of event conditions used during the generation of the event, in particular on the type of information contained in the event, i.e. whether the event comprises information on signs (traffic signs), information on the magnitude of the temporal contrast (difference between light and dark areas of the image) or time difference, and / or information on the spatial contrast or spatial difference. An image may be reconstructed by processing a group of events together and updating the reconstructed image based on the processed group of events. Examples of methods for imaging from events are disclosed in [1]. They may be used to reconstruct images from events even when an event-based camera captures a flickering light source.

[0016] Then, a region or regions are determined within the reconstructed image where flickering light sources that may be associated with portions of the spatial scene of interest are likely to be present, and a subset of pixel parameter sets that are associated with the determined region or regions are then determined, where association means that a change in pixel parameters within the subset of pixel parameters includes or implies a change in the imaging behavior of pixels within the determined region or regions. When the event-based camera is further operated, suitable second values ​​are determined or assigned to the subset of parameters that reduce flicker in the determined region(s). The second value of the pixel parameter associated with the pixels within the determined region may be different for each pixel within the determined region. The pixel parameter affecting the imaging behavior of pixels not within the at least one determined region may be set to optimize, for example, one or both of a signal-to-noise ratio (SNR) and a temporal resolution. In this way, only the imaging behavior of the pixels involved in the flicker is adapted (changed to suit the conditions) to reduce the flicker, while the remaining pixels are adapted to optimize their overall imaging behavior based on the specific requirements to be achieved, for example, to achieve a minimum signal-to-noise ratio.

[0017] In one embodiment of the method according to the present invention, an event condition included in the generation of the plurality of events is embodied as a temporal contrast condition, a temporal difference condition, a spatial contrast condition, and / or a spatial difference condition.

[0018] In a further embodiment of the method according to the invention, the accumulation of the analog domain detection signal of each pixel (provided by a photodetector belonging to each of the pixels) is influenced by at least one exposure time parameter (using one exposure time parameter associated with a pixel the duration of the exposure time may be controlled and set), and each pixel or group of pixels of the plurality of pixels is associated with an exposure time parameter of the at least one exposure time parameter.

[0019] In a further embodiment of the method according to the invention, the pixels are arranged in a pixel array having a plurality of rows and a plurality of columns, and the event-based camera comprises a readout processor which, during operation of the event-based camera, successively reads out one row after the other from the plurality of rows to generate a plurality of events.

[0020] In a further embodiment of the method according to the invention, the image is i) Multiple events and, optionally, ii) at least one image of a previously generated image when the generated events are generated based on one or both of a temporal contrast condition and a temporal difference condition; is reconstructed based on If the events contain intensity information, an image can be reconstructed based on the events alone. If the events contain contrast or difference information, an image can be reconstructed based on multiple events and a previously generated image.

[0021] In a further embodiment of the method according to the invention, the step of determining at least one region in the reconstructed image comprises: a) determining at least one histogram for a reconstructed image; b) thresholding the reconstructed image based on the determined at least one histogram (thresholding results in two thresholded reconstructed images); c) filtering noisy bright pixels in the two thresholded reconstructed images using at least one morphological function; d) determining bright spots in the two thresholded reconstructed images using a blob (a collection of pixels with the same density) detection technique, and selecting bright spots in the two thresholded reconstructed images from the determined bright spots that are substantially similar in shape, the selected bright spots being at least one of the determined regions; Equipped with.

[0022] One or more histograms may be determined from the reconstructed image. If multiple histograms are determined, for example, different bin sizes may be used for the different histograms. One or two thresholds may be used to threshold the reconstructed image: (i) In the case of one threshold and two histograms, for example, two reconstructed images each thresholded with two thresholds may be determined by thresholding the two histograms each with one threshold. (ii) In the case of two thresholds (which may be different from each other) and one histogram, for example, two reconstructed images each thresholded with two thresholds may be determined by thresholding one histogram with the two thresholds. The threshold(s) used in the thresholding process may be determined based on the histogram or alternatively may be provided in advance. Morphological functions (e.g., erosion, dilation, opening (first erosion, then concatenation of dilations), closing (first dilation, then concatenation of erosion), proper opening, proper closing, or automedian) may be used. Such morphological functions may be appropriately combined to filter out bright and dark noise pixels from the two thresholded reconstructed images. Bright spots are determined in the two thresholded reconstructed images using well-known algorithms for blob detection. Such bright spots are compared across the two thresholded reconstructed images, and the bright spots may be used to determine at least one region that is substantially found within the two thresholded reconstructed images. In an alternative embodiment, only one thresholded reconstructed image is determined and compared to the reconstructed image to determine suitable bright spots. Besides using a histogram and a threshold, the at least one region may alternatively be determined as follows: Gamma correction with two different power parameters may be employed to provide two different gamma-corrected reconstructed images, which may then be processed as described for the two thresholded reconstructed images.

[0023] In a further embodiment of the method according to the invention, the step of determining at least one region in the reconstructed image comprises applying a trained neural network to the reconstructed image, the trained neural network being at least one bounding box feature; at least one anchor box configuration; the morphology of the segmentation mask; a form of probability heatmap; and at least one region is provided as output. The reconstructed images, which could be provided as input to the neural network, may be embodied as grayscale or color images of any bit depth for each of the image channels. Optionally, a neural network may receive the histogram as input. The neural network may include multiple layers, some of which may be embodied as convolutional layers. The neural network may also include one or more nonlinear activation functions and / or normalization layers. Some of the layers of the neural network may have skip connections between layers. Bright areas are manually annotated in the form of bounding boxes, and the neural network may be trained using one or both sets of annotated reconstructed images and their corresponding histograms. The annotations may all contain the same label (e.g., "light"), or they may distinguish between different types of bright areas (e.g., "car lights," "street lights," etc.). Prior to training, the neural network may be initialized with random numbers and trained from scratch, or several layers may be pre-trained using publicly available classification datasets.

[0024] In a further embodiment of the method according to the invention, the determined subset of pixel parameters comprises at least one exposure time parameter and at least one digital accumulation parameter, and assigning second values ​​to the at least one exposure time parameter and the at least one digital accumulation parameter comprises: a) assigning or receiving a required total accumulation period, the required total accumulation period being based on a flicker frequency of at least one flickering light source that is active during operation of the event-based camera; b) determining a second value for at least one exposure time parameter based on the determined required or desired total accumulation period and sampled detection signals of pixels in the determined at least one region obtained during operation of the event-based camera, such that pixels in the determined at least one region are not oversaturated during further operation; c) assigning a second value for at least one digital accumulation parameter based on the determined second value for the at least one exposure time parameter and the determined required or desired total accumulation period; Equipped with.

[0025] The first value of the exposure time parameter may be determined to maximize the signal-to-noise ratio (SNR) of the pixel while simultaneously preventing pixel saturation. One of the main noise sources in an image sensor is photon shot noise, where the SNR is approximately equal to the square root of the total number of photons collected during one exposure. Therefore, it may be desirable to have a long exposure time as long as the pixel is not saturated. However, the longer the exposure time, the lower the pixel's sampling rate, and therefore the temporal resolution. In general, there is a trade-off between pixel SNR and temporal resolution, and the optimum point typically depends on the application: for example, for automotive applications, the temporal resolution must be at least 30 Hz, and in this case the exposure time cannot be set longer than 33 ms even if the SNR is compromised. On the other hand, optimizing temporal resolution can mean setting the exposure time as short as possible given the required SNR for the application. Because SNR is approximately the square root of the integrated signal, it is possible to estimate the SNR by evaluating the integrated signal. For example, if the application requires an SNR of 100, which corresponds to an integrated signal of 500, the exposure time could be set to the shortest value that results in an integrated signal of at least 500.

[0026] The first value of the digital accumulation parameter may be set to "1" to maximize the temporal resolution of the pixel, i.e., the digital accumulation includes or accounts for only one sampled detection signal. Alternatively, the first value of the digital accumulation parameter may be set to achieve a certain desired total accumulation period that corresponds to a desired temporal resolution of the pixel based on the application, for example, a desired temporal resolution of 30 Hz corresponds to a desired total accumulation period of approximately 33 ms. In particular, the first value of the digital accumulation parameter may be derived as the desired total accumulation period divided by the first value of the exposure time parameter.

[0027] Flicker mitigation can be achieved directly by modifying the exposure time parameter(s), the second value of which may be set equal to or greater than the total required accumulation period, as long as the pixel is not saturated. To confirm or project whether a pixel is saturated or likely to become saturated, the sampled detection signal of that pixel may be compared to a predetermined potential saturation threshold. If the sampled detection signal corresponds to, for example, a 10-bit value, the potential saturation threshold may be set to a value smaller than the maximum 10-bit value, such as 500 or 700. For example, if the required total accumulation period is 16 ms, the sampled detection signal obtained in the previous 16 ms can be evaluated to predict whether the second value of the exposure time parameter can be set to 16 ms without saturating the pixels. Alternatively, the second value of the exposure time parameter can be immediately set to 16 ms and then readjusted if the sampled detection signal subsequently indicates pixel saturation (e.g., the second value of the exposure time parameter can be reduced, e.g., by 50%, if the sampled detection signal exceeds a potential saturation threshold).

[0028] If setting the second value of the exposure time parameter for the required total accumulation period would saturate the pixel, then flicker mitigation is achieved with the help of digital accumulation parameter(s), where the second value of the exposure time parameter may be set to a value equal to or higher than the first value, i.e., the highest value that does not saturate the pixel. For example, if the second value of the exposure time parameter is set to 8 ms to prevent pixel saturation, and the total accumulation period required is 16 ms, then the second value of the digital accumulation parameter would be set to "2".

[0029] In general, the second value of the digital accumulation parameter may be derived as the required total accumulation period divided by the second value of the exposure time parameter, or, if there is a desired total accumulation period set by the application, the second value of the digital accumulation parameter may be derived by dividing the higher value between the required total accumulation period and the desired total accumulation period by the second value of the exposure time parameter.

[0030] Preferably, both the exposure time parameter and the required total accumulation duration may be set as the square of the previously defined interval (corresponding to the minimum exposure time). Setting the exposure time and the required total accumulation duration in this way will simplify the derivation of the digital accumulation parameters and the subsequent normalization of the digitally accumulated sampled detection signal.

[0031] Overall, given that first values ​​of exposure time parameters and digital accumulation parameters for a pixel have been determined to optimize one or both of the SNR and temporal resolution of the pixel based on the application, the following steps may be performed for determining second values ​​of exposure time parameters and digital accumulation parameters for pixels within the determined at least one region. First, it is checked whether the first value of the exposure time parameter is longer than the total required accumulation period. If so, a second value of the exposure time parameter and the digital accumulation parameter may be set equal to their respective first values. If not, it is checked whether the second value of the exposure time parameter can be set to the total required accumulation period without saturating the pixels. If so, set the second value of the exposure time parameter to the total cumulative period required. If not possible, the second values ​​of the exposure time parameters are set to at least the respective first values ​​or to a value higher than the respective first values ​​without saturating the pixels. Second, after the second value of the exposure time parameter is determined, a second value of the digital accumulation parameter is determined based on the second value of the exposure time parameter and the total required or desired accumulation period. These steps may preferably be performed by a readout processing device.

[0032] Therefore, the second values ​​of the exposure time parameter and the digital accumulation parameter may vary between pixels. Pixels within the determined at least one region may have pixel parameters that achieve flicker reduction as a first priority goal and optimize one or both of SNR and temporal resolution based on the application as a second priority goal. Meanwhile, pixels not within the determined at least one region may have pixel parameters that optimize one or both of SNR and temporal resolution based on the application as the only priority goal.

[0033] In a further embodiment of the method according to the invention, the flicker frequency is assumed to be greater than 50 Hz (value frequency), in particular greater than 80 Hz (value frequency), and the required total accumulation period is determined such that flicker is reduced in reconstructed images based on multiple events generated during further operation of the event-based camera. Alternatively, the flicker frequency is assumed to be greater than 50 Hz, in particular greater than 80 Hz, and the required total accumulation period is selected to be greater than the reciprocal of the assumed flicker frequency, such that flicker is reduced in a reconstructed image based on multiple events generated during further operation (6) of the event-based camera. The reciprocal of the assumed flicker frequency may correspond to the flicker period of the flickering light source.

[0034] Since the exposure time parameter and the digital accumulation parameter are determined and therefore known, the digitally accumulated sampled detection signal can be normalized based on this knowledge. For example, the digitally accumulated sampled signal can be normalized by determining the total number of intervals corresponding to the digital accumulation. Many flickering light sources in a scene of interest, such as a road network, may be assumed to have a known flicker frequency. When the method according to the present invention is used in such a scene of interest, the flicker frequency is known and therefore does not need to be determined, for example. For example, 50 Hz and 80 Hz correspond to the lowest flicker frequencies encountered in automotive use cases. These frequencies and higher are typically imperceptible to the human eye and therefore need to be mitigated by the event-based camera in order to be correctly interpreted.

[0035] In a further embodiment of the method according to the invention, each pixel in the plurality of pixels has its own exposure time parameter, or groups of pixels in the plurality of pixels share one exposure time parameter.

[0036] Additionally, each pixel in the plurality of pixels may have its own digital accumulation parameter, or groups of pixels in the plurality of pixels may share a digital accumulation parameter.

[0037] In a further embodiment of the method according to the invention, at least one of the flickering light sources is embodied as one or both of a traffic light light source and a vehicle light source.

[0038] In a further embodiment of the method according to the invention, the events are generated at multiples of a period of a world clock of the event-based camera, in particular the events may be generated at boundaries of predefined intervals and the exposure time may be an integer multiple of the interval.

[0039] In a second aspect of the present invention, a) an event-based camera having at least i) a plurality of pixels arranged in a pixel array; ii) a read processing unit; iii) a parameter memory for storing a pixel parameter set; an event-based camera comprising at least b) a computing device (particularly embodied as a microprocessor); An assembly is provided comprising: The event-based camera is adapted to transmit generated events to the computing device, the assembly being adapted to implement the method according to the first aspect of the invention. The event-based camera is adapted to perform steps 1), 4) and 5) of the method according to the first aspect of the present invention, and the computing device is adapted to perform steps 2) and 3) of the method according to the first aspect of the present invention.

[0040] In one embodiment of the assembly according to the second aspect of the invention, the computing device is part of the event-based camera, or the computing device is an external computing device separate from the event-based camera.

[0041] In a third aspect, the present invention provides a vehicle comprising an assembly according to the second aspect, the vehicle being adapted to travel in a scene of interest comprising traffic signals and / or lights of other vehicles.

[0042] The vehicle may be embodied as, for example, a conventional car or a motorcycle.

[0043] Exemplary embodiments of the invention are disclosed in the following description and illustrated in the drawings. [Brief explanation of the drawings]

[0044] [Figure 1] FIG. 1 illustrates a schematic diagram of one embodiment of a method for mitigating flickering light in a scene of interest captured by an event-based camera. [Figure 2]FIG. 2 is a schematic diagram of an event-based camera imaging a scene of interest with a traffic light as a flickering light source. [Figure 3] FIG. 3 is a schematic diagram of an event-based camera imaging a scene of interest including a car with at least one car light as a flickering light source. [Figure 4] FIG. 4 shows a schematic diagram of an embodiment of an assembly according to the invention. DETAILED DESCRIPTION OF THE INVENTION

[0045] 1 illustrates a schematic diagram of one embodiment of a method for reducing flickering light in a scene of interest captured by an event-based camera. The event-based camera is operated to image the scene of interest, and the imaging outputs a plurality of events 2. The event-based camera comprises a plurality of pixels whose imaging operation is affected by a set of pixel parameters. The event-based camera is operated by assigning a first value to the set of pixels. Based on the plurality of events 2, an image is reconstructed 3, and at least one region is determined 4 in the reconstructed image. The region corresponds to a spatial region in the scene of interest and potentially comprises a flickering light source. Thereafter, a subset of pixel parameters of the set of pixel parameters is determined 5, the subset of pixel parameters being associated with pixels of the determined at least one region, and a second value is determined 5 for the subset of pixel parameters, the second value being determined to reduce flicker for corresponding pixels of the determined at least one region. Prior to further operating the event-based camera using the updated pixel parameters, the first value of the subset of pixel parameters is updated 5 and replaced 5 with the determined second value of the subset of pixel parameters. In this way, flicker in the determined at least one region is reduced in a reconstructed image based on the further operating.

[0046] 2 shows a schematic diagram of an event-based camera 7 imaging a scene of interest that includes a traffic light source as a flickering light source. The event-based camera 7 may be mounted on or installed in, for example, a vehicle configured for autonomous driving, and the event-based camera 7 may therefore image the scene of interest through which the vehicle is traveling to provide information to a driving algorithm that steers the vehicle. In the scene of interest, traffic light sources 8, which comprise light-emitting diodes, may be controlled by pulse-width modulated signals, which induce flickering. Using the method according to the present invention, such light flickering is mitigated by an appropriate combination of algorithmic processing and hardware settings in the event-based camera 7.

[0047] Figure 3 shows a schematic diagram of an event-based camera imaging a scene of interest in which a motor vehicle having at least one car light is present as a flickering light source. Figure 3 is structurally similar to Figure 2. In Figure 3, the flickering light source is embodied as a car light 9 of a further vehicle, and the event-based camera 7, as in Figure 2, may be mounted on a vehicle traveling through a scene of interest in which the further vehicle, e.g., a vehicle having at least one car light, is also traveling.

[0048] 4 shows a schematic representation of one embodiment of an assembly according to the present invention, comprising an event-based camera 7 and a computing device 10. The computing device 10 is adapted to reconstruct an image from a plurality of events generated by the event-based camera 7 imaging a scene of interest, and to determine at least one region in the reconstructed image. The computing device 10 receives as input a plurality of events from the event-based camera 7 and provides as output coordinates of pixels within the determined at least one region to the event-based camera 7, the event-based camera 7, specifically the readout processing device of the event-based camera, determines second values ​​of the subset of pixel parameters associated with the pixels within the determined at least one region, and the event-based camera 7 then updates its respective hardware parameters. 4, the computing device is physically separate from the event-based camera, for example located in the trunk of the car the assembly is installed in. Alternatively, the computing device 10, typically embodied as a microprocessor, may be part of the event-based camera 7, meaning that all processing takes place within the event-based camera 7.

Claims

1. 1. A method for reducing flickering light in a scene of interest imaged with an event-based camera (7), comprising: The event-based camera (7) comprises a plurality of pixels; The imaging behavior of the plurality of pixels is influenced by a pixel parameter set comprising at least one exposure time parameter and / or at least one digital accumulation parameter, and the method comprises: 1) operating an event-based camera (7) with assigning a plurality of first values ​​to the pixel parameter sets, operating the event-based camera (7), the operation (1) providing a plurality of events (2) generated by the event-based camera (7); 2) reconstructing (3) an image based at least on the generated plurality of events (2); 3) determining (4) at least one region in the reconstructed image, wherein the determined at least one region corresponds to at least one spatial region in the scene of interest comprising at least one flickering light source (8, 9) active during the operation of the event-based camera (7); determining (4) at least one region within the reconstructed image; 4) determining a subset of pixel parameters (5) of the set of pixel parameters, the subset of pixel parameters affecting the imaging behavior of a plurality of pixels within the determined at least one region; assigning (5) second values ​​to the determined subset of pixel parameters, the second values ​​being determined to reduce flicker within the determined plurality of pixels in the at least one region; and (5) updating the first values ​​assigned to the subset of pixel parameters using the second values; and 5) further operating (6) the event-based camera (7) using the updated pixel parameter set; A method comprising:

2. 2. The method of claim 1, wherein an event condition included in generating the plurality of events (2) is embodied as at least one of a temporal contrast condition, a temporal difference condition, a spatial contrast condition, and a spatial difference condition.

3. The accumulated analog domain detection signal of each pixel is affected by at least one exposure time parameter, and the detection signal is provided by one photodetector provided in the corresponding pixel; The method of claim 1 or 2, wherein each pixel of the plurality of pixels is associated with one exposure time parameter of the at least one exposure time parameter.

4. the plurality of pixels are arranged in a pixel array having a plurality of rows and a plurality of columns; The event-based camera (7) is equipped with a readout processing device, 4. The method according to claim 1, wherein the readout processing device is adapted to successively read out one row after another from the plurality of rows during operation of the event-based camera (7) and to generate a plurality of events.

5. The image is i) a plurality of events (2); ii) at least one previously generated image when the generated plurality of events (2) are generated based on one or both of a temporal contrast condition and a temporal difference condition; The method according to any one of claims 2 to 4, wherein the reconstructed (3) is based on

6. The step of determining (4) at least one region in the reconstructed image comprises: a) determining at least one histogram for the reconstructed image; b) thresholding the reconstructed image based on the determined at least one histogram, wherein said thresholding results in two thresholded reconstructed images; c) filtering noisy bright pixels in the two thresholded reconstructed images using at least one morphological function; d) determining bright spots in the two thresholded reconstructed images using a blob detection technique, and selecting from the determined bright spots a plurality of bright spots in the two thresholded reconstructed images that are substantially similar in shape, the selected plurality of bright spots being the determined at least one region; The method of any one of claims 1 to 5, comprising:

7. the step of (4) determining the at least one region in the reconstructed image comprises applying a trained neural network to the reconstructed image; the trained neural network at least one bounding box feature; at least one anchor box configuration; the morphology of the segmentation mask; a form of probability heatmap; and providing the at least one region as an output.

8. the determined subset of pixel parameters comprises at least one exposure time parameter and at least one digital accumulation parameter; The step of (5) assigning second values ​​to the at least one exposure time parameter and the at least one digital accumulation parameter comprises: a) determining a required total accumulation period, the required total accumulation period being based on a flicker frequency of the at least one flickering light source (8, 9) operating during operation (1) of the event-based camera (7); b) assigning (5) a second value for at least one exposure time parameter based on the determined required or desired total accumulation period and sampled detection signals of pixels in the determined at least one region obtained during operation (1) of the event-based camera (7) so as to prevent oversaturation of pixels in the determined at least one region during further operation (6); c) assigning a second value for at least one digital accumulation parameter based on the determined second value for the at least one exposure time parameter and the determined required or desired total accumulation period; The method of any one of claims 3 to 7, comprising:

9. The flicker frequency is assumed to be greater than 50 Hz, in particular greater than 80 Hz, 9. The method of claim 8, wherein the required total accumulation period is selected to be greater than the inverse of the assumed flicker frequency to reduce flicker in reconstructed images based on multiple events generated during further operation (6) of the event-based camera (7).

10. each pixel in said plurality of pixels having its own exposure time parameter; or 10. The method of claim 8 or 9, wherein multiple groups of pixels within the plurality of pixels share an exposure time parameter.

11. 11. The method according to any one of claims 1 to 10, wherein at least one of the flickering light sources (8, 9) is embodied as one or both of a traffic light source (8) and an automobile light source (9).

12. 12. The method according to any one of claims 1 to 11, wherein the plurality of events (2) are generated at multiples of a period of a world clock of the event-based camera (7).

13. a) an event-based camera (7) comprising at least i) a plurality of pixels arranged in a pixel array; ii) a readout processing device; iii) a parameter memory for storing pixel parameter sets; an event-based camera (7) comprising at least b) a computing device (10), in particular embodied as a microprocessor; An assembly (7, 10) comprising: The event-based camera (7) transmits the generated events (2) to the computing device (10), The assembly (7, 10) is adapted to carry out the method according to any one of claims 1 to 12, The event-based camera (7) is adapted to perform steps 1), 4) and 5) of the method, The assembly (7, 10) is adapted to perform steps 2) and 3) of the method.

14. the computing device (10) is part of the event-based camera (7), or 14. The assembly (7, 10) according to claim 13, wherein the computing device (10) is an external computing device separate from the event-based camera (7).

15. 15. A vehicle comprising an assembly according to claim 13 or 14, the vehicle being adapted to travel through a scene of interest comprising a plurality of traffic signals (8) and a plurality of lights (9) of other vehicles.

Images