Apparatus for sensor signal processing

Abstract

It is art objective of the present invention to provide a signal processing apparatus. In more detail, it is the purpose of the described invention to remove fixed pattern noise from the output of the shutterless sensor array signals and thus allow high quality image and make possible spatial signal processing such as convolution and neighborhood magnitude comparison. It is a further objective of the signal processing apparatus to allow a simple to implement method for fixed pattern noise removal to be used with shutterless sensors. It is a further objective of the signal processing apparatus to make possible a fast way to remove the fixed pattern noise with shutterless sensors.

Description

[0001] Apparatus for sensor signal processing

[0002] TECHNICAL FIELD OF THE INVENTION

[0003] The present invention relates to a signal processing apparatus according to the preamble of the appended independent claim. The invention also relates to a signal processing system comprising a plurality of such apparatuses.

[0004] The project leading to this application has received funding from the European Union's Horizon 2020 research and innovation programme under grant agreement No 101016734.

[0005] BACKGROUND OF THE INVENTION

[0006] Shutterless sensors are common in thermal imaging and have also gained popularity in visual range imaging since the introduction of event cameras. The emphasis on the prior art of the event generating devices has been on monitoring the sensor output and indicating changes in the sensor output as change events. Typically the event sensors that are used are shutterless, i.e., they are not reset but the sensor output is available at any time. This is in contrast to integrating sensors such as a three or four transistor image sensor, where the signal is first reset and then allowed to aggregate for a period of time. The benefit of an integrating sensor is that it allows to use so called double sampling or correlated double sampling in order to eliminate much of the fixed pattern noise and reset noise from the signal.

[0007] Shutterless sensors on the other hand allow detecting changes in the signal faster since they do not have the dead time of reset followed by integration. The downside is that since they are not reset, no double sampling or correlated double sampling is possible, and the signal exhibits large fixed pattern noise due to readout electronics introduced mismatch, for example shutterless image sensors provide a low quality image. Consequently, it is not practical to directly compare values of sensor outputs in an array of shutterless sensors for the uncompensated representations. It is the purpose of the described invention to remove fixed pattern noise from the output of the shutterless sensor array signals and thus allow high quality image and make possible spatial signal processing such as convolution and neighborhood magnitude comparison. Low frequency noise components such as flicker noise can also be suppressed. Currently there are no published means to correct the fixed pattern noise of shutterless event camera pixels and although there exists prior in the thermal imaging, e.g., US10735680822, these methods are slower to converge and more complex to implement than the present invention, employing operations such movement extraction. The present invention can be applied to the thermal fixed pattern removal as well.

[0008] OBJECTIVES OF THE INVENTION

[0009] It is the main objective of the present invention to reduce or even eliminate the prior art problems presented above.

[0010] It is an objective of the present invention to provide a signal processing apparatus for processing signals provided by shutterless sensors that provide at least two dimensional topologically arranged data. In more detail, it is an objective of the invention to provide a signal processing apparatus that determines a fixed pattern noise compensation term based on signals corresponding to neighborhood signals, and the compensation can be performed adaptively while the sensor data is used for imaging or image analysis tasks. It is a further objective of the invention to provide a signal processing apparatus that can perform convolution-like operations for fixed pattern compensated signals within the neighborhood, and carry out threshold operation on the convolution-like operation. Also, it is a further objective of the invention to provide a signal processing system comprising a plurality of signal processing apparatuses and use the signal processing system for processing shutteriess sensor data in applications like image analysis.

[0011] In order to realize the above-mentioned objectives, the signal processing apparatus according to the invention is characterised by what is presented in the characterising portion of the appended independent claim. Advantageous embodiments of the invention are described in the dependent claims.

[0012] SUMMARY OF THE INVENTION

[0013] According to a first aspect of the invention there is provided a signal processing apparatus that comprises: means to receive a first scene change signal from a first scene change indicator, a first signal channel (101a) arranged to provide an apparatus first signal corresponding to a spatial location L_A, a second signal channel (101b) arranged to provide an apparatus second signal corresponding to a spatial location L_B, and a third signal channel (101c) arranged to provide an apparatus third signal corresponding to a spatial location L_C, wherein spatial location L__A is in the neighborhood of spatial locations L_B and L__C, and the apparatus first signal is from a shutterless image sensor, configured to reduce fixed pattern noise from the apparatus first signal and to provide a resulting first compensated signal to a first output channel (107), a compensation memory (104) having a first digital memory (1040) for storing a first compensation term, with an input and an output, wherein the compensation memory has means to store data provided to the input, and means to provide the stored data to the output, a first means to sum (103) having a first input, a second input and a first output channel (107), wherein o the first input is connected to the first signal channel (101) and arranged to receive the apparatus first signal, o the second input is connected to compensation memory (104) and arranged to receive a first compensation term, o the first means to sum computes the sum of the first input and the second input and provides the resuiting first compensated signal to the first output channel (107), an arithmetic circuit (106) configured to receive the first compensated signal from the first output channel (107), the apparatus second signal from the second signal channel (101b) and the apparatus third signal from the third signal channel (101c) and arranged to compute a first weighted sum and to provide the result to an output of the arithmetic circuit (106), a second means to sum (105) arranged to receive the first scene change signal (130), and having a first input, a second input and an output of the second means to sum, wherein o the first input is arranged to receive the first weighted sum from the output of the arithmetic circuit (106), o the second input is arranged to receive the first compensation term from the compensation memory (104), o when the first scene change signal (130) is indicating scene change, the second means to sum computes the sum of the first input and the second input and provides the result to the output of the second means to sum, o when the first scene change signal (130) is indicating no scene change, the second means to sum provides the second input of the second means to sum to the output of the second means to sum, o the output of the second means to sum is connected to the input of the compensation memory (104) and arranged to provide the second sum value to the compensation memory (104).

[0014] According to an embodiment of the signal processing apparatus comprises: the second means to sum (105) further comprises means to limit the first weighted sum to values between a first negative limit and a first positive limit and provides a first limited sum, wherein o when the first scene change signal (130) is indicating scene change, the second means to sum (105) provides the first limited sum to the output of the second means to sum o when the first scene change signal (130) is indicating no scene change, the second means to sum (105) provides the second input of the second means to sum to the output of the second means to sum. According to an embodiment of the signa! processing apparatus comprises: the second means to sum (105) further comprises means to quantize the first limited sum to at least three quantization levels and provides a first quantized sum, wherein o when the first scene change signal (130) is indicating scene change, the second means to sum (105) provides the first quantized sum to the output of the second means to sum o when the first scene change signal (130) is indicating no scene change, the second means to sum (105) provides the second input of the second means to sum to the output of the second means to sum.

[0015] According to an embodiment of the signal processing apparatus comprises: the compensation memory (104) having a second digital memory (1041) for storing a second compensation term, means to select which of the first and second digital memories (1040, 1041) is written, and means to select which of the first and second digital memories (1040, 1041) is arranged to the output of the compensation memory.

[0016] According to an embodiment of the signal processing apparatus comprises: an output memory (120) arranged to receive input from the first output channel (107) and provide output to an output memory channel (121).

[0017] According to an embodiment of the signal processing apparatus comprises: a threshold circuit (108) arranged to accept the output of the arithmetic circuit (106) as the input and to determine whether the absolute value of output of the arithmetic circuit is larger than a first threshold value, and in such a case generate a signal indicating an event El and provide the event indication signal to an event channel (109).

[0018] According to an embodiment of the signal processing apparatus comprises: a fourth signal channel and an apparatus fourth signal corresponding to a spatial location L_D, wherein spatial location L_D is in neighborhood of spatial location L_A an appended arithmetic circuit (106b) arranged to receive signals from the first output channel, the second input channel, the third input channel and the fourth input channel (107, 101b, 101c, lOld) and arranged to provide a weighed sum to the output of the appended arithmetic circuit (106b). According to an embodiment of the signal processing apparatus comprises: means to vary the weights of the weighted sum computation within the arithmetic circuit: at a first time instant arrange a first, set of weights to the arithmetic unit (106;106b) at a second time instant arrange a second set of weights to the arithmetic unit (106, 106b)

[0019] According to an embodiment of the signal processing apparatus comprises: means to determine whether the absolute value of the output of the arithmetic circuit is larger than a second threshold value, and in such a case generate a signal indicating an event E2 and provide an event indicating signal to event channel (109)

[0020] According to an embodiment of the signal processing apparatus comprises: an event memory (110) with an input and an output, with means to store data provided to the input, and means to provide the stored data to the output, wherein the event channel (109) is arranged to the input and the output is arranged to the input of an event arithmetic circuit (111) an event arithmetic circuit (111) with a first input and a second input arranged to provide a third event Fl to an event memory output channel (112).

[0021] According to an embodiment of the signal processing system comprises: at least three signal processing apparatuses (100) according to any of the preceding claims: a first signal processing apparatus (100a), a second signal processing apparatus (100b), a third signal processing apparatus (100c) according to any of the preceding claims, and a signal source 201 arranged to provide input data to the signal processing apparatuses (100a, 100b, 100c).

[0022] According to an embodiment of the signal processing system comprises: the signal generation apparatuses (100a, 100b, 100c) that use different thresholds values.

[0023] The exemplary embodiments of the invention presented in this text are not interpreted to pose limitations to the applicability of the appended claims. The verb "to comprise" is used in this text as an open limitation that does not exclude the existence of also unrecited features. The features recited in the dependent claims are mutually freely combinable unless otherwise explicitly stated.

[0024] BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The novel features which are considered as characteristic of the invention are set forth in particular in the appended claims. The invention itself, however, both as to its construction and its method of operation, together with additional objects and advantages thereof, will be best understood from the following description of specific embodiments when read in connection with the accompanying drawings.

[0026] Fig. la,b Illustrates neighborhood locations relevant to the invention,

[0027] Fig. 2 Illustrates a signal processing apparatus according to a first embodiment of the invention,

[0028] Fig. 3. Illustrates a flowchart of an exemplary way to use the signal processing apparatus,

[0029] Fig. 4 Illustrates an embodiment of the signal processing apparatus where the compensation memory has two memory locations.

[0030] Fig. 5 Illustrates an embodiment of the signal processing apparatus that comprises an output memory,

[0031] Fig. 6 Illustrates an embodiment of the signal processing apparatus that comprises a threshold circuit,

[0032] Fig. 7 Illustrates an embodiment of the signal processing apparatus with a fourth signal channel,

[0033] Fig. 8 Illustrates an embodiment of the signal processing apparatus comprising an event memory and an event arithmetic circuit.

[0034] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS OF THE INVENTION

[0035] In order to provide fixed pattern correction in shutterless sensors new means to process and store event signals are introduced in this disclosure. In one embodiment of the invention a shutterless image sensor in visible or beyond visible wavelength range is used to provide the input signal.

[0036] In other embodiments, shutterless audio, tactile, olfactory, radar or any other sensor type that is arranged to provide at least two-dimensional topographically arranged data can be used. The term pixel used throughout this application refers to a device that provides a signal from a shutterless image sensor on one or more of the sensor types listed above or sensors sensitive to any other modality.

[0037] FIXED-PATTERN NOISE COMPENSATION METHOD AND VARIATIONS

[0038] Let coordinates x and y denote the locations of shutterless pixels in a two-dimensional sensor data array, and let us denote the digitized output of a pixel at location (x, y) at time t by p(x, y, t), i.e., a digital signal produced by a shutterless pixel at location (x, y). Later in the text this signal is referred to as an apparatus first signal. Let us further suppose that T is a set of consequent time points (or iteration steps) corresponding to digital outputs of a considered pixel, and that | T| denotes the number of such time paints, and finally that the following approximately holds for each shutterless pixel near a particular operating point:

[0039] In other words, we assume that the digitized outputs of each shutterless pixel are distributed around a mean value b(x, y), which can be different for each pixel. These mean values comprise of the contribution of the mean light intensity measured by the pixels at these locations and the fixed-pattern noise.

[0040] In the following we describe a method by which the values b(x, y) can be effectively transformed to be equal for each pixel. An underlying assumption is that the camera is moving in a dynamic scene so that the mean of the light intensity measured in the vicinity (neighborhood) of each pixel is approximately constant. To achieve the unifying transformation of b(x, y), we add a first compensation term c(x, y) to the apparatus first signal. Now for each pixel,

[0041] Let us assume that the outputs of pixels are obtained at sampling rate / -- ™ where At is the sampling time interval. Let us define the neighborhood N(x, y) for each pixel as some subset of pixel locations (x,> y<) whose distance from pixel at location fx, y) is at most d, where d is some fixed constant. For example, the value for d may be d ~ 1, which corresponds to the set of neighboring locations in cardinal directions. This is illustrated in Fig. la) where pixel locations L_B and L_C are in the first neighborhood (d=l) of pixel location L_A. In an alternative embodiment, d = 1.5, which corresponds to a 3 x 3 patch around the considered location. Yet another alternative is illustrated in Fig. lb) where L_B and L_C are from the third neighborhood of I. .A. Throughout in this application when we say that a signal corresponds to a spatial location, we mean that the signal originates from a two-dimensional array of sensor elements (for example shutterless pixels, photon detectors, microphones, tactile sensors and so on) and the locations of the sensor elements are denoted by e.g., L__A, L_B and L C. Let us first assume that the neighborhood N(x, y) does not contain the location (x, y) itself, and denote the number of iocations in N(x, y) by | N(x, y)\ .

[0042] For each iteration step t we adjust the compensation term c(x, y) (here we explicitly denote the iteration step by t) by the foilowing rule: where ꭤ(t) > 0 is a scaling parameter, and tprevdenotes the previous iteration step. The purpose of this adjustment is to gradually change the terms c(x, y) so that eventually the sum c(x, y) + b(x, y) is approximately constant in a local neighborhood. Since b(x, y) depends on the fixed-pattern noise value at location (x, y), this procedure largely removes the spatial variance of the fixed pattern noise. Note that Eqn. 4 can be computed as a weighted sum of terms of type

[0043] Typically, in the beginning of the iterative adjustment of the compensation term, the scaling parameter would be higher in comparison to the end of the iterative adjustment process. For example, in the beginning, the adjustment term could be 1 / 2 while in the end of the iterative adjustment process, the adjustment term could be 1 / 16. A larger adjustment term leads to faster but more coarse-grained convergence of the compensation term, while a smaller adjustment term achieves a slower but a more accurate compensation term.

[0044] Mathematically, when there is a need to stop adjusting the compensation term, the scaling parameter can be assigned to zero. Preferably, the entire compensation term adjustment process is halted when the compensation term is not updated. Furthermore, when the fixed pattern noise compensation method is ongoing, the method works also when the scaling parameter is kept at a fixed value, for example 1 / 16. This way, a slow but accurate compensation term adjustment is achieved.

[0045] In a preferred embodiment of the fixed pattern noise compensation method it is assumed that the input scene is changing so tha t local gradients average out during the multiple rounds of iterative update process of the compensation term. Therefore, the compensation term is only updated when there is an indication that the input scene is changing, and the compensation term is not updated (update process is halted) when there is an indication that the scene is not changing. Halting the update process based on indication of scene change is explained below (Fig. 3).

[0046] In an alternative embodiment of the invention, Δc(x,y, t) is limited (clipped) to a first limited sum Δc(x,y, t) that obtains values within a preferably symmetric range, so that the first limited sum has values between a first negative limit and a first positive limit (e.g., -1 / 16 and 1 / 16). The first positive and negative limits can be chosen freely while understanding that their magnitudes have an effect on convergence rate and stability. In a yet further embodiment of the invention, Δc(x,y, t) is quantized resulting in a first quantized sum Ac(x,y, t). In one embodiment of the invention, the first quantized sum is quantized to three levels: the first negative limit, 0 and the first positive limit. For example, Ac(x,y, t) may obtain values -1 / 16, 0 and 1 / 16. However, the method works with different numbers of quantization levels. In these embodiments of the invention either Δc(x,y, t) or Δc(x,y, t) can be applied in Eqn. 3 instead of Δc(x,y, t).

[0047] Fig. 2 illustrates a signal processing apparatus (100) intended for removing fixed pattern noise from input signal provided by a shutterless sensor. The signal processing apparatus receives via a first signal channel (101a) the apparatus first signal which is denoted as p(x,y,t) in equations 1, 2, and 4, corresponding to spatial location L_A. Additionally, the signal processing apparatus receives the apparatus second signal corresponding to spatial location L_ Bb via a second digital channel (101b), and the apparatus third signal corresponding to spatial location L_C via a third digital channel (101c). The signal processing apparatus (100) also comprises a first compensation memory (104) having a first digital memory (1040) for storing a first compensation term and providing the first compensation term c(x,y) to the output of the compensation memory (104).

[0048] The signal processing apparatus also comprises a first means to sum (103), having a first, input. receiving the apparatus first signal p(x,y,t) and the second input receiving the compensation term c{x,y) from the compensation memory (104) and provides the sum of p(x,y,t) and c(x,y), denoted as a first compensated signal to a first output channel (107). The first means to sum can be a dedicated summing circuit, or any other means to sum that is made available to the signal processing apparatus (100). For example, the first means to sum could use the same summing resources as an arithmetic circuit (106). Such a broad definition of the word "means" applies throughout this application.

[0049] The signal processing apparatus (100) further comprises an arithmetic circuit (106) arranged to receive the apparatus first, signal, the apparatus second signal and the apparatus third signal, the arithmetic circuit arranged to provide a first weighted sum of the apparatus first signal, the apparatus second signal and the apparatus third signal resulting in a first output of the arithmetic circuit (106). The first output of the arithmetic circuit corresponds to Eqn. 4. An example of a weighted sum is the apparatus first signal subtracted by the average of the apparatus second and third signals scaled by a scaling parameter a(t).

[0050] The signal processing apparatus (100) further comprises a second means to sum (105), accepting a first input from the output of the arithmetic circuit (106) and the second input from the output of the compensation memory (104), and provides an output to the input of the compensation memory (104). Furthermore, the second means to sum receives an indication of scene change via a first scene change signal (130). The scene change signal can be obtained e.g., by using an inertial measurement unit, accelerometer, or by carrying out scene motion analysis.

[0051] The output of the second means to sum (105) depends on the first scene change signal: when the first scene change signal is indicating scene change, the second means to sum computes the sum of the first input and the second input and provides the result to the output of the second means to sum, when the first scene change signal is indicating no scene change, the second means to sum provides the second input of the second means to sum to the output of the second means to sum.

[0052] Fig. 3 frustrates an exemplary embodiment the method of using the signal processing apparatus (100).

[0053] The idea is to compute a first compensation term that counteracts the fixed pattern noise

[0054] ■Fixed pattern noise originates for example from device mismatch

[0055] ■ Temperature affects the fixed pattern noise

[0056] ■Fixed pattern noise also alters with time (flicker noise)

[0057] Because of the above,, the compensation terms need to be updated regularly.

[0058] The method is based on the assumption that the average of the neighborhood closely approximates the desired center pixel value

[0059] We use at least two neighborhood signals to compute the first compensation term. We select the neighbor locations L_B and L_C such that they are not collinear with respect to the center location L..A. Such a choice of neighbor locations allows fast convergence of compensation terms in a two-dimensional array, while avoiding learning global features of illumination patterns. o Our fixed pattern noise compensation method assumes that that the input scene is changing so that local gradients average out during the compensation

[0060] ■ As an example, the input scene is changing when a shutterless image sensor is rotated or moved and thus tend to produce different signals in consecutive frames. ■ Whether or not input scene is changing is indicated with the scene change signal (130). For example, the scene change signal can be based on a global measure and shared by multiple signal processing apparatuses in a system comprising a plurality of signal processing apparatuses. Alternatively, the scene change signal can be based on local scene change. ■ When input scene is indicated to be changing, and the compensation term update is ongoing, continue iterating the first compensation term.

[0061] ■When input scene is not changing as indicated by the scene change signal (130), or the compensation term update is turned off, use the first compensation term stored in the compensation memory o Real local gradients vanish (average to zero) over multiple frames when the input scene is changing

[0062] 8Hence, as is important in mobile sensing applications, compensation can be carried out on a moving platform. There is no restriction in the speed at which the scene is changing, i.e., the method works at high and low scene changing speeds. If the scene is not changing, the first compensation term is not updated which prevents the apparatus from over-adapting to scene gradients. o Compensation can be carried out on the fly (when new data keeps flowing)

[0063] 8This allows compensation for time-varing changes in the sensor such as temperature and flicker noise

[0064] The method illustrated in Fig. 3 is a flow chart of how the signal processing apparatus of Fig. 2 could be used. Note that the apparatus first digital input is uncompensated (contains fixed pattern noise), whereas the apparatus second and third inputs can be either compensated on uncompensated.

[0065] It is possible that (Eqn. 1) (approximately) holds only when the pixel digital outputs are within a sub-interval / of all possible values, due to e.g. readout circuit nonlinearities. In other words, whenever each pixel value p(x, y, t) E l. This means that there should be multiple compensation terms cfx, y, !) for different pixel value ranges. In Fig. 4 this is arranged by allocating multiple memory locations (first digital memory 1040, second digital memory 1041) into the compensation memory. When multiple compensation terms are used, the level of the signals (apparatus first input signal, apparatus second input signal and / or apparatus third signal) can be used to determine which compensation term from the first digital memory is chosen.

[0066] Another example of utilizing more than one memory location in the compensation memory is that the signal processing apparatus is used in a time-multiplexed manner to compensate signals from multiple locations. One example is an image sensor that would have one signal processing apparatus per a column of pixels, arranged to provide compensation signals for the whole column. In this case there would be a need to have memory locations in the compensation term at least the amount of pixels that are in the column.

[0067] If the compensation memory in the signal processing apparatus has multiple memory locations, there needs to be a controller that controls which memory location (1040, 1041) is written and which memory location (1040, 1041) is read

[0068] If the signal processing apparatus (100) serves signals from multiple locations, there need be switches that select from which neighborhood the input signals are provided to the signal processing apparatus When the signal processing apparatus serves multiple locations (e.g., columns) and the apparatus second and third signals are compensated (fixed pattern noise removed), there needs to be a memory for storing first compensated signals of the neighborhood. For example, when compensation terms are computed row-by-row, first compensated signals of previously compensated rows would not be available without a memory Fig. 5 illustrates the signal processing apparatus comprising an output memory (120) arranged to receive input from the digital output channel (107) and to provide an output of the output memory to an output memory channel (121)

[0069] COMPUTING A THRESHOLDED WEIGHTED SUM OF PIXEL OUTPUTS

[0070] Let us consider a neighborhood N(x, y) of location (x, y) in a two-dimensional sensor data array, where the location (x, y) may be contained in N(x, y). Let us assume that the first digital signal p(x, y, t) has been corrected by adding the compensation term c(x, y) (and thus in the following we use shorthand notation pc(x, y, t) to mean p(x, y, t) + cfx, yj).

[0071] Let wix^fxi, y,) denote digital weights, where each weight corresponds to a location (Xj, y>) G iV(.x,y). Note that the apparatus second and third signals need to be compensated (fixed pattern noise has been removed) if the weights corresponding to these terms are nonzero.

[0072] Fig. 6 illustrates a signal processing apparatus (100) comprising a threshold circuit (108) arranged to accept the output of the arithmetic circuit (106) as the input and to determine whether the absolute value of the output of the arithmetic circuit is larger than a first threshold value, and in such a case generate a signal indicating an event El and provide the event indication signal to an event channel (109)

[0073] Using the element (106) depicted in Fig. 6 it is possible to compute the weighted sum which is thresholded by the element (108), yielding the spatial events El(x, y, t) by the rule where 9min(x, y, t) and 9max(x>y> 0a rethe minimum and maximum threshold values, respectively at location (x, y) at time t. In Fig. 6 the threshold circuit receives as input a weighted sum of the first compensated signal, the apparatus second input and the apparatus third signal.

[0074] In many cases it is useful to have more than three inputs to the signal processing apparatus. Fig. 7 illustrates a signal processing apparatus comprising an arithmetic circuit that is arranged to receive an apparatus fourth input via a fourth signa! channel. Naturally, the signal processing apparatus could be expanded to accept more than four inputs.

[0075] In practice, the weights can correspond for example to gradient computation kernels such as the Haar mother wavelet, the Sobel filter, or the Scharr filter. The weights could be fixed or they could be alterable. The simplest implementation would have only very few nonzero fixed weights. Also, typically integer arithmetic is easy to implement. For example, division of an integer by a power of two reduces to a shift of the bits of the integer and such operations are thus preferred.

[0076] The thresholds can be changed as the function of time or space: for example, all locations can be thresholded using the same threshold value, but this value can be changed with time according to some rule, or all locations can have different but fixed threshold values. Multiple spatial events corresponding to the same iteration step can be generated at a given location by using multiple threshold values. For example, spatial events E2(x,y,t) can be generated by rule

[0077] Let tpreVand t be successive iteration steps corresponding to the output of the event El at location (x, y). In some cases the successive events need to be memorized. Fig. 8 illustrates the signal processing apparatus (100) comprising

[0078] An event memory (110) with an input and an output, with means to store data provided to the input, and means to provide the stored data to the output, wherein the event channel (109) is arranged to the input and the output is arranged to the input of an event arithmetic circuit (111) An event arithmetic circuit (111) with a first input and a second input is arranged to provide a third event Fl to the event memory output channel (112)

[0079] The event memory (110) and the event arithmetic circuit (111) can be used to generate spatiotemporal events. The concept of spatiotemporal events Fl(x, y, t), illustrated by the output 112 in Fig. 8, is obtained as a function of El(x, y, t) and El(x, y, tprev). A possible rule for generating the spatiotemporal event Fl{x, y, t) is given below (highlighted in gray):

[0080] In the above table whenever the spatial event changes there is a spatiotemporal event generated that can distinctively express the mode of change. It is also possible to code the spatiotemporal events in another manner, where the spatiotemporal event type denotes the direction of change. Such coding is shown below

[0081] It is also possible in some cases, e.g. simple neighbor difference between neighboring pixels, to code the spatial events as positive events, i.e. there is a gradient above threshold, where the negative spatial events of one location are interpreted as positive spatial events in the other direction. In this case, the spatiotemporal events can be generated by the following simplified coding

[0082] A useful feature of spatial and spatiotemporal events ~ in contrast to conventional temporal events ~ is that they do not respond to changes in global illumination such as the flickering of the lighting. This also makes it possible to gate conventional temporal events using spatial or spatiotemporal events. For example, if Q(x, y, t) represents conventional temporal difference events, one can obtain gated temporal events Q(x,y, t) as or where ix| denotes the absolute value of x. In effect, if the neighborhood gradients are not changing, there is no response. Corner paints can be defined as certain patterns of spatial events in local neighborhoods. For example, in a 3 x 3 neighborhood, when the neighboring locations of (xi, yi) are traversed in clockwise fashion, if there are more than 4 successive positive or negative spatial events (thresnolded differences of signals of center location and neighboring locations of the eight nearest neighbors), a corner event is produced at (x, y). This resembles the definition of the Features from Accelerated Segment Test (FAST) corner.

[0083] Fig. 9. Illustrates and example embodiment of corner event generation. Here white blocks represent positive spatial events and black blocks represent negative spatial events. Striped boxes represent zero events. Left: two examples of patterns yielding a corner event are presented. Right: two examples of patterns not yielding a corner event are presented.

[0084] In another embodiment of the invention the corner is extracted using signals from a larger neighborhood than the nearest neighbors.

[0085] If corners are tracked, they can be used to compute optical flow by estimating the movement of the corners. Furthermore, if spatial events are obtained from an image pyramid (a set of different resolution representations of the pixel outputs), the corners can be used to estimate optical flow, where motions of features related to different spatial frequencies can be estimated as motions of corner points at different resolutions. Another application of image pyramid -based spatial events is that they can be used as inputs of different levels of convolutional neural networks (where deeper levels of the network take lower resolution frames of spatial events as inputs).

[0086] Fig 10 illustrates a signal processing system comprising at least three signal processing apparatuses a first signal processing apparatus, a second signal processing apparatus, a third signal processing apparatus and a digital signal source 201 arranged to provide input signals to the signal processing apparatuses (101a, 101b, 101c). In this example the signal processing apparatuses' first output channels are arranged to the neighboring signal processing apparatuses.

[0087] Fig. 11 illustrates a signal processing system comprising at least three signal processing apparatuses comprising at least three signal processing apparatuses a first signal processing apparatus, a second signal processing apparatus, a third signal processing apparatus

[0088] And an digital signal source 201 arranged to provide data to the signal processing apparatuses (101a, 101b, 101c). The signal processing apparatuses in Fig. 11 have output memory (120), and the corresponding output memory channels as well as first output channels are arranged to the inputs of the neighboring signal processing apparatuses. Note that the signal processing apparatuses in a signal processing system can use different adaptation values corresponding to different spatial locations.

Claims

CLAIMS1. A signal processing apparatus (100), comprising: means to receive a first scene change signal from a first scene change indicator, a first signal channel (101a) arranged to provide an apparatus first signal corresponding to a spatial location L_A, a second signal channel (101b) arranged to provide an apparatus second signal corresponding to a spatial location L_B, a third signal channel (101c) arranged to provide an apparatus third signal corresponding to a spatial location L_C, wherein spatial location L_A is in the neighborhood of spatial locations L_B and L_C, and the apparatus first signal is from a shutterless image sensor, configured to reduce fixed pattern noise from the apparatus first signal and to provide a resulting first compensated signal to a first output channel (107), characterized in that the apparatus (100) comprises: a compensation memory (104) having a first digital memory (1040) for storing a first compensation term, with an input and an output, wherein the compensation memory has means to store data provided to the input, and means to provide the stored data to the output, a first means to sum (103) having a first input, a second input and a first output channel (107), wherein o the first input is connected to the first signal channel (101) and arranged to receive the apparatus first signal, o the second input is connected to compensation memory (104) and arranged to receive a first compensation term, o the first means to sum computes the sum of the first input and the second input and provides the resulting first compensated signal to the first output channel (107), an arithmetic circuit (106) configured to receive the first compensated signal from the first output channel (107), the apparatus second signal from the second signal channel (101b) and the apparatus third signal from the third signal channel (101c) and arranged to compute a firstweighted sum and to provide the result to an output of the arithmetic circuit (106), a second means to sum (105) arranged to receive the first scene change signal (130), and having a first input, a second input and an output of the second means to sum, wherein o the first input is arranged to receive the first weighted sum from the output of the arithmetic circuit (106), o the second input is arranged to receive the first compensation term from the compensation memory (104), o when the first scene change signal (130) is indicating scene change, the second means to sum computes the sum of the first input and the second input and provides the result to the output of the second means to sum, o when the first scene change signal (130) is indicating no scene change, the second means to sum provides the second input of the second means to sum to the output of the second means to sum, o the output of the second means to sum is connected to the input of the compensation memory (104) and arranged to provide the second sum value to the compensation memory (104).

2. The signal processing apparatus (100) according to claim 1, characterized in that the signal processing apparatus (100) comprises: the second means to sum (105) further comprises means to limit the first weighted sum to values between a first negative limit and a first positive limit and provides a first limited sum, wherein o when the first scene change signal (130) is indicating scene change, the second means to sum (105) provides the first limited sum to the output of the second means to sum o when the first scene change signal (130) is indicating no scene change, the second means to sum (105) provides the second input of the second means to sum to the output of the second means to sum.

3. The signal processing apparatus (100) according to claim 2, characterized in that the signal processing apparatus (100) comprises: the second means to sum (105) further comprises means to quantize the first limited sum to at least three quantization levels and provides a first quantized sum, wherein o when the first scene change signal (130) is indicating scene change, the second means to sum (105) provides the first quantized sum to the output of the second means to sum o when the first scene change signal (130) is indicating no scene change, the second means to sum (105) provides the second input of the second means to sum to the output of the second means to sum.

4. The signal processing apparatus (100) according to any of the preceding claims, characterized in that the signal processing apparatus (100) comprises: the compensation memory (104) having a second digital memory (1041) for storing a second compensation term, means to select which of the first and second digital memories (1040, 1041) is written, and means to select which of the first and second digital memories (1040, 1041) is arranged to the output of the compensation memory.

5. The signal processing apparatus (100) according to any of the preceding claims, characterized in that signal processing apparatus (100) further comprises: an output memory (120) arranged to receive input from the first output channel (107) and provide output to an output memory channel (121).

6. The signal processing (100) according to claims 1, 2, 3 or 4, characterized in that signal processing apparatus (100) further comprises: a threshold circuit (108) arranged to accept the output of the arithmetic circuit (106) as the input and to determine whether the absolute value of output of the arithmetic circuit is larger than a first threshold value, and in such a case generate a signal indicating an event El and provide the event indication signal to an event channel (109).

7. The signal processing apparatus (100) according to claims 1, 2, 3 or 4, characterized in that signal processing apparatus (100) further comprises: a fourth signal channel and an apparatus fourth signal corresponding to a spatial location L D, wherein spatial location L_D is in neighborhood of spatial location L_A an appended arithmetic circuit (106b) arranged to receive signals from the first output channel, the second input channel, the third input channel and the fourth input channel (107, 101b, 101c, 101d) and arranged to provide a weighed sum to the output of the appended arithmetic circuit (106b).

8. The signal processing apparatus (100) according to any of the preceding claims, characterized in that the arithmetic circuit (106, 106b), comprises means to vary the weights of the weighted sum computation within the arithmetic circuit: at a first time instant arrange a first set of weights to the arithmetic unit (106, 106b), at a second time instant arrange a second set of weights to the arithmetic unit (106, 106b).

9. The signal processing apparatus (100) according to claim 6, characterized in that the threshold circuit (108) further comprises means to determine whether the absolute value of the output of the arithmetic circuit is larger than a second threshold value, and in such a case generate a signal indicating an event E2 and provide an event indicating signal to event channel (109).

10. The signal processing apparatus (100) according to any of claims 6, 8, 9, characterized that the signal processing apparatus (100) further comprises: an event memory (110) with an input and an output, with means to store data provided to the input, and means to provide the stored data to the output, wherein the event channel (109) is arranged to the input and the output, is arranged to the input of an event arithmetic circuit. (Ill) an event arithmetic circuit (111) with a first input and a second input arranged to provide a third event Fl to an event memory output channel (112).

11. A signal processing system characterized in that the signal processing system comprises at least three signal processing apparatuses (100) according to any of the preceding claims: a first signal processing apparatus (100a), a second signal processing apparatus (100b), a third signal processing apparatus (100c) according to any of the preceding claims, and a signal source (201) arranged to provide input data to the signal processing apparatuses (100a, 100b, 100c).

12. The event generation system according to claims 11, characterized in that, the signal generation apparatuses (100a, 100b, 100c) use different thresholds values.

Images