Image sensor system, electronic device and method for operating an image sensor system

Abstract

An image sensor system (1) is provided that comprises a pixel array (10) with a plurality of pixels (12), each of the pixels (12) comprising a photosensitive stage (14) configured to generate signal values. It further comprises a readout circuit (20) coupled to the pixel array (10) and being configured to read out and digitize the signal values from a subset of pixels (12) of the pixel array (10). It further comprises a programmable image filter (30) coupled to the readout circuit (20), the programmable image filter (30) comprising a memory (32) having stored a coefficient matrix, the programmable image filter (30) being configured to apply the coefficient matrix onto the digitized signal values (DSV) to generate and output filtered signal values (FSV). In addition, an electronic device (100) and a method for operating an image sensor system (1) is provided.

Description

[0001] Description

[0002] IMAGE SENSOR SYSTEM, ELECTRONIC DEVICE AND METHOD FOR OPERATING AN IMAGE SENSOR SYSTEM

[0003] The present application relates to an image sensor system . It further relates to an electronic device comprising the image sensor system and to a method for operating an image sensor system .

[0004] In applications such as augmented reality (AR) , virtual reality (VR) , robotics , and computer vision, CMOS image sensor ( CIS ) cameras can be used to monitor the surrounding environment . The images taken by CIS cameras are processed to detect obj ects in the scene and track obj ects across multiple scenes in time . Currently, CIS cameras only provide images while image filtering and computer vision functions , algorithms take place / run on user hardware / sof tware . Computer vision algorithms process image data by performing a certain set of basic filtering operations like edge detection, low- pass / high-pass filtering, image convolution, image sharpening, etc . Optical flow is one of the most basic operations / algorithms used in computer vision for obj ect detection and tracking . Data outputs of CIS cameras , i . e . , images and videos , are fed into complex hardware and software to identi fy and track obj ects with among the first steps being to be performed to compute optical flow . Current systems require high power consumption and computations in external hardware and software components which is not useful for low-power embedded systems . At least one obj ect of particular embodiments is to provide a compact , solid-state image sensor for providing image data and filtered image data .

[0005] This obj ect is achieved with the subj ect-matter of the independent claims . Embodiments and developments derive from the dependent claims .

[0006] According to at least one embodiment , an image sensor system is provided . The image sensor system comprises a pixel array with a plurality of pixels , each of the pixels comprising a photosensitive stage configured to generate signal values .

[0007] The photosensitive stage may comprise a photodiode . The photodiode converts electromagnetic radiation received by the pixel into electrical signals corresponding to the signal values . The plurality of signal values from the pixels may be used to reconstruct an image or video of a scene captured by the pixel array . Further elements like a pixel buf fer and a trans fer gate coupled between the photodiode and an input of the pixel buf fer could also be included in each of the pixels . The pixel array may be implemented as rolling or global shutter pixel array . Image sensor system may be reali zed as CMOS image sensor, CIS .

[0008] According to at least one embodiment , the image sensor system comprises a readout circuit coupled to the pixel array and being configured to read out and digiti ze the signal values from a subset of pixels of the pixel array .

[0009] The subset of pixels may include all pixels or a group of pixels in the pixel array, for example one or several rows of the pixel array, or a region of interest (ROI ) comprising of a smaller subset of pixels from among the entire pixel array . That the readout circuit is configured to digiti ze the signal values can mean that the readout circuit comprises an analog- to-digital converter, ADC . The readout circuit may further comprise a buf fer, the buf fer being configured to temporarily store the digiti zed signal values .

[0010] According to at least one embodiment , the image sensor system comprises a programmable image filter coupled to the readout circuit , the programmable image filter comprising a memory having stored a coef ficient matrix, the programmable image filter being configured to apply the coef ficient matrix onto the digiti zed signal values to generate and output filtered signal values .

[0011] The programmable image filter may be implemented as or may comprise a matrix multiplication and / or matrix convolution unit . In particular, such matrix multiplication and / or matrix convolution unit can be hardware implemented . Matrix multiplication / convolution on hardware is disclosed, for example , in Asgari et al . , "MEISSA: Multiplying Matrices Ef ficiently in a Scalable Systolic Architecture" , 2020 IEEE 38th International Conference on Computer Design ( ICCD) , Hartford, CT , USA, 2020 , pp . 130- 137 , the disclosure content of which is hereby incorporated by reference . The programmable image filter is arranged downstream of the readout circuit . The pixel array, the readout circuit , and the programmable image filter may be integrated on the same chip . The programmable image filter may comprise two inputs . A first input is configured to receive the digiti zed signal values from the readout circuit . A second input is configured to receive a user defined coef ficient matrix . The filtering function of the programmable image filter depends on the coefficient matrix stored in the memory. Thus, the image filter is programmable and depends on a user input. The image sensor system may comprise one or more programmable image filters. In case that multiple programmable image filters are comprised in the image sensor system, the programmable image filters may be arranged in parallel or in series. Multiple programmable image filters may perform different filter operations, as explained below in context of image gradients.

[0012] That the coefficient matrix is applied onto the digitized signal values can mean that a matrix representation of the digitized signal values is multiplied with the coefficient matrix. Applying the coefficient matrix may also include a convolution operation and / or matrix multiplication both of which use multiply-accumulate operations. Convolution and matrix multiplications can be implemented in hardware using dedicated matrix / vector multiplication blocks used in conjunction with accumulate and control logic blocks, also known as multiply addition units.

[0013] By applying the coefficient matrix, the signal values are filtered. Thus, the filtered signal values are based on the original digitized signal values. In some implementations, the applied coefficient matrix represents a clear filter, such that the filtered signal values correspond to the original digitized signal values. In this case, the filtered signal values are pure image data, and the digitized signal values are effectively unfiltered. It is also possible that unfiltered signal values, i.e. pure image data, are output by the image sensor system in parallel to the filtered signal values. The filtered signal values may also be referred to as filtered image data. The memory of the programmable image filter can be on-chip or of f-chip . For example , the memory is reali zed as EEPROM or flash memory . In some implementations , more than one coef ficient matrix is stored in the memory of the programmable image filter, or di f ferent coef ficient matrices are uploaded to the memory one after the other . The di f ferent coef ficient matrices may be applied successively to the digiti zed signal values to generate intermediate filtered signal values and final filtered signal values .

[0014] According to at least one embodiment , an image sensor system is provided . The image sensor system comprises a pixel array with a plurality of pixels , each of the pixels comprising a photosensitive stage configured to generate signal values . It further comprises a readout circuit coupled to the pixel array and being configured to read out and digiti ze the signal values from a subset of pixels of the pixel array . It further comprises a programmable image filter coupled to the readout circuit , the programmable image filter comprising a memory having stored a coef ficient matrix, the programmable image filter being configured to apply the coef ficient matrix onto the digiti zed signal values to generate and output filtered signal values .

[0015] The proposed concept is based on the following considerations , among others :

[0016] The present disclosure provides a solution for a compact , solid-state , image sensor system with outputs for image data and filtered image data, such as edges within an image . The image sensor system outputs the various data modes ( image and / or filtered image ) using the same readout circuit and pipeline for all the data streams . The image sensor system allows for calculation of filtered image data, e . g . edges within the image , using the programmable image filter with the choice of the coef ficient matrix . Further, the image sensor system allows for low power operation as the filtering calculations are done at pixel level thereby reducing the need for hardware and / or software post-processing . The simultaneous output of image data and filtered image data from the same image sensor allows to simpli fy the system and allows to quickly identi fy and track obj ects within the scene . The output of the image sensor system can either be the image or a filtered version of the image . Advantageously, performing image filtering can be performed on-chip as soon as the image is captured . Further, the coef ficient matrix for image filtering is user selectable and ( re- ) programmable . The hardware implementation allows for complex image filtering and computer vision algorithms . These complex algorithms can be implemented as multiple , successive , matrix multiplications . Since the filter is implemented as hardware next to the sensor array, the total operation can be quick, of fers power savings , and reduces system complexity for the end user .

[0017] According to at least one embodiment , the programmable image filter is configured to be programmed by selecting at least one coef ficient matrix stored in the memory .

[0018] For example , in a first programming state , a first coef ficient matrix is selected to perform a first filtering operation on the digiti zed signal values . In a second programming state , a second coef ficient matrix is selected to perform a second filtering operation on the digiti zed signal values . Thus , the programmable image filter can be configured and reconfigured . Using the programmable image filter different filtering operations can be performed by selecting respective coefficient matrices. Thus, a plurality of different filtering operations can be implemented with the same hardware.

[0019] According to at least one embodiment, the memory is configured to store a unity matrix, such that the programmable image filter can be programmed as clear filter for outputting pure image data of an image captured by the pixel array. This can mean that by applying the unity matrix onto the digitized signal values (e.g. multiplying) , the digitized signal values are effectively unchanged, resulting in pure image data to be output. However, it is also possible that the image data is bypassed as additional output parallel to the filtered image data. In that case, the image data may be less noisy, since there is no matrix operation that potentially adds noise to the image data.

[0020] In addition, or alternatively, the memory is configured to store a gradient matrix, in particular Prewitt, Sobel, Krisch and / or Laplace matrices, such that the programmable image filter can be programmed as gradient detector for outputting gradients along lateral directions of an image captured by the pixel array. In other words, by applying one or more of the above-mentioned matrices on the digitized signal values (e.g. by convolution, multiply-accumulate operations) , edges within the image are detected. For example, Prewitt Matrices for vertical (Gx) and horizontal (Gy) edge detection can look like : As another example , Sobel Matrices for vertical ( Gx) and hori zontal ( Gy) edge detection can look like :

[0021] Both directions can be applied in each case . In some implementations , the image sensor system comprises more than one programmable image filter . For example , a first image filter stores in its memory a coef ficient matrix for hori zontal edge detection, and a second image filter stores in its memory a coef ficient matrix for vertical edge detection . The first and the second image filter may form the gradient detector or may form separate gradient detectors .

[0022] The matrices may be called kernels . The kernels may be applied to a matrix representation of the signal values of the subset of pixels . For every signal value within the matrix representation, a multiply-accumulate operation can be performed involving the values in the kernel and the signal values and its surroundings to determine the value for the filtered signal value . The result of the operation are edge data within the signal values of the selected subset of pixels , which can be assembled to edge data of the full image accordingly . In this case , the edge data are the respective filtered signal values .

[0023] In at least one embodiment , the memory is configured to store a Gaussian matrix, such that the programmable image filter can be programmed as Gaussian image filter . By using said Gaussian image filter, smoothing of the image data can be performed . In particular, smoothing of the image data can be performed before edge detection described above . For example , a Gaussian image filter in case of a 5x5 kernel (M) can look like :

[0024] In addition, or alternatively, the memory is configured to store an inverse of a pixel point spread function in matrix form, such that the programmable image filter can be programmed to function as a filter that compensates the optical crosstalk between pixels . This can mean that an inverse of a pixel Point Spread Function ( PSF) is stored, such that the programmable image filter can be programmed as pixel crosstalk correction filter by which the crosstalk caused due to light leakage from pixels caused by non-random def ects / predictable reasons / design choices such as partial deep trench isolation between pixels , etc . , are compensated by loading the inverse of measured or calculated pixel PSF from memory . Thus , crosstalk between pixels of pixel array can be corrected .

[0025] In addition, or alternatively, the memory is configured to store an image sharpness matrix, such that the programmable image filter can be programmed as image sharpness improvement filter . Thus , an MTF (modulation trans fer function) sharpness improvement can be applied to the image data . In other words , the sharpness of the image can be improved . Compensating the PSF, as explained above , may improve sharpness , too . However, contrary to compensating the PSF, an image sharpness matrix may be a mathematically constructed matrix, which is not based on the knowledge of any design details of the pixel ( such as partial deep trench, etc . ) .

[0026] According to at least one embodiment , each of the pixels further comprises a sample-and-hold stage configured to temporarily store the signal values .

[0027] The sample-and-hold stage , S / H stage , is coupled to the photosensitive stage . For example , the S / H stage is coupled to a di f fusion node of the photosensitive stage via a first source follower . The S / H stage may comprise one or more than one storage element , in particular a capacitor . The S / H stage may further comprise other components like switches to trans fer the signal values to the storage elements . I f the pixels comprise a S / H stage , they may be reali zed as global shutter pixels , in particular voltage-domain global shutter pixel . This means that the signal values are stored in voltage domain . Advantageously, the signal values can be stored at pixel level . However, the pixels may also be implemented as charge domain global-shutter pixels .

[0028] According to at least one embodiment , the sample-and-hold stage of each pixel comprises a first storage element configured to store a first signal value generated during a first integration period and a second storage element configured to store a second signal value generated during a second integration period .

[0029] The second integration period may follow the first integration period . The integration periods refer to time periods in which the photodiode of the photosensitive stage accumulate charge carriers by conversion of electromagnetic radiation . The first integration period and the second integration period may have a same duration . The first storage element may be reali zed as first capacitor that is coupled to the photosensitive stage via a first switch . The second storage element may be reali zed as second capacitor that is coupled to the photosensitive stage via a second switch . The first storage element and the second storage element may be arranged cascaded . In that case , signal values may be distributed between the storage elements . After the first integration period, the first signal value is transmitted to a di f fusion node of the photosensitive stage by activating a trans fer switch that is coupled between the photodiode and the di f fusion node . Then, switches of the S / H stage , e . g . the first switch and the second switch, may be activated to trans fer the first signal value to the first storage element where it is stored in voltage domain . After the second integration period, the second signal value is transmitted to the di f fusion node by activating the trans fer switch again . Then, switches of the S / H stage , e . g . the second switch, may be activated to trans fer the second signal value to the second storage element where it is stored in voltage domain . Therefore , at least two signal values of the same " frame" can be stored at pixel level .

[0030] According to at least one embodiment , the sample-and-hold stage of each pixel further comprises a third storage element configured to store a first reset level generated during the first integration period and a fourth storage element configured to store a second reset level generated during the second integration period .

[0031] The third storage element may be reali zed as third capacitor that is coupled to the photosensitive stage via a third switch . The fourth storage element may be reali zed as fourth capacitor that is coupled to the photosensitive stage via a fourth switch . The third storage element and the fourth storage element may be arranged cascaded . In that case , the reset levels may be distributed between the storage elements . The first reset level is trans ferred to the third storage element before the trans fer switch is activated at the end of the first integration period . The second reset level is trans ferred to the fourth storage element before the trans fer switch is activated at the end of the second integration period . The reset levels may correspond to noise during the first and the second integration period, respectively . The first signal value may comprise noise that is correlated to the first reset level . The second signal value may comprise noise that is correlated to the second reset level . Thus , using the first and the second reset level correlated double sampling can be performed during readout .

[0032] Alternatively, the sample-and-hold stage of each pixel does not comprise the third and the fourth storage element for storing reset levels . It may only comprise the first and the second storage element for storing signal values . It that case , correlated double sampling cannot be performed but double delta sampling is still possible to consider fixed pattern noise of the pixels .

[0033] According to at least one embodiment , the sample-and-hold stage of each pixel comprises a first capacitor bank and a second capacitor bank arranged in parallel , wherein the first capacitor bank comprises a first capacitor as first storage element connected in cascade with a second capacitor as second storage element , and wherein the second capacitor bank comprises a third capacitor as third storage element connected in cascade with a fourth capacitor as fourth storage element . The capacitor banks are coupled to the photosensitive stage . For example , they are coupled to the di f fusion node via the first source follower . Thus , an ef ficient architecture of the sample-and-hold stage is provided to store signal values and reset levels at pixel level .

[0034] According to at least one embodiment , the sample-and-hold stage of each pixel comprises at least four storage elements or exactly four storage elements . In particular, the sample- and-hold stage comprises the first , the second, the third, and the fourth storage element as mentioned above . Thus , no further storage elements are required .

[0035] According to at least one embodiment , the image sensor system is configured to output the digiti zed signal values and the filtered signal values in parallel .

[0036] According to at least one embodiment , the image sensor system further comprises a calculation circuit coupled to the readout circuit and configured to , for each pixel of the subset of pixels , determine a signal velocity value based on a di f ference between the first signal value and the second signal value , and on a duration of the second integration period .

[0037] The calculation circuit may be arranged in parallel to the programmable image filter . The calculation circuit receives from the readout circuit the first signal value and the second signal value generated by each pixel . Thus , the calculation circuit can determine the signal velocity value for each pixel or for each pixel within the subset of pixels . The calculation circuit outputs the signal velocity values . Since each pixel may store two signal values , i . e . the first and the second signal value, a signal velocity can be determined at pixel / sensor level.

[0038] According to at least one embodiment, the programmable image filter is programmed as gradient detector for an image captured by the pixel array, and wherein the gradient detector together with the calculation circuit form an optical flow engine integrated into the image sensor system.

[0039] In other words, the image sensor system can simultaneously output image data and all the data required to calculate optical flow, in particular a gradient of the image in time (dl / dt) , a gradient of the image along a horizontal direction (dl / dx) , and a gradient of the image along a vertical direction (dl / dy) . The gradient of the image in time, dl / dt, may also called velocity map. The image sensor system outputs the various data modes using the same readout circuitry and pipeline for all the data streams. It allows for calculation of the image gradients along lateral directions using the programmable image filter, for example programmable matrix multiplication units. It allows for low power operation, as the calculations of gradients and velocity are done at the pixel / sensor level thereby eliminating the need for postprocessing software. The simultaneous output of image data and data required to calculate the optical flow from the same image allows to simplify the system and to quickly identify and track objects within the scene. The data outputs use the same readout circuitry and same pipeline for all output data modes (image, velocity (dl / dt) , gradients (dl / dx, dl / dy) ) while keeping small image sensor package footprint and reducing the need for external components. The calculation of the output data modes is done simultaneously, in hardware (on the CIS chip, at pixel level) and no software ( on external processor ) postprocessing is required . The benefit of in-pixel simultaneous calculation of data modes results in overall system performance to be improved, require less computation, and improve overall system power consumption . This is of advantage for battery powered applications such as drones , mobiles , AR / VR, robotics , computer vision, etc . A further advantage is having velocity and image gradient output modes integrated on-chip . This is reali zed by per pixel capacitor banks and a velocity computation block ( calculation circuit ) on-chip . Multiple chip stacking can be used to help in tightly integrating all the above features into a compact package with small footprint .

[0040] According to at least one embodiment , the programmable image filter and the calculation circuit are arranged in parallel . Thus , the calculation of signal velocity values and the calculation of filtered signal values , in particular edge data of the image , can be performed simultaneously . Therefore , the processing speed is increased, and the signal velocity values , and the filtered signal values can be processed separately .

[0041] According to at least one embodiment , the readout circuit comprises at least one analog-to-digital converter coupled at its input side to the pixel array and at its output side to a buf fer of the readout circuit , the buf fer being configured to temporarily store the digiti zed signal values from the subset of pixels and to provide them to the programmable image filter .

[0042] The analog-to-digital converter, ADC, is electrically connected to outputs of the pixels . In various implementations several pixels may share a common ADC, such that a number of ADCs employed in the image sensor system is smaller than the number of pixels . In some implementations , a separate ADC may be provided for each pixel or for each row or column of pixels . The buf fer is implemented as memory element that stores a plurality of digiti zed signal values . For example , the buf fer is configured to store the digiti zed signal values from one or more rows of the pixel arrays . I f only a single ADC is used to read data from the pixel array, the implementation is simple , the speed of the system is however reduced . I f a multitude of ADCs is used to read data from the pixel array, speed is improved at the cost of increased circuit complexity .

[0043] According to at least one embodiment , applying the coef ficient matrix comprises a multiply-accumulate operation with the coef ficient matrix and a matrix representation of the digiti zed signal values . The multiply-accumulate operation may include matrix multiplication and / or convolution . In general , in the multiply-accumulate operation two numbers are multiplied and the product is added to an accumulator . The multiply-accumulate operation may be implemented in hardware . Thus , the programmable image filter may be implemented as , or may comprise a multiplieraccumulator unit or multiple such units .

[0044] According to at least one embodiment , the programmable image filter further comprises a storage unit and a multiply- accumulate circuit , wherein the multiply-accumulate circuit is configured to : receive the digiti zed signal values from the storage unit and a first coef ficient matrix from the memory; apply the first coef ficient matrix onto the digiti zed signal values to generate intermediate filtered signal values ; store the intermediate filtered signal values in the storage unit ; receive the intermediate filtered signal values from the storage unit and a second coef ficient matrix from the memory; and apply the second coef ficient matrix onto the intermediate filtered signal values to generate further intermediate or final filtered signal values .

[0045] Therefore , consecutive matrix operations on the same image are enabled . The inclusion of the storage unit in the programmable image filter enables the image sensor system to perform complex filtering algorithms such as edge detections involving several steps , e . g . two sequential matrix operations using two di f ferent coef ficient matrices . For example , a process of edge / gradient detection can comprise two steps both of which are matrix multiply-accumulate operations . First , image smoothing is performed using a Gaussian smoothing matrix followed by edge detection using a di f ferentiation matrix such as a Sobel matrix . Thus , the edge detection algorithm can be implemented in hardware by inclusion of a storage unit in the programmable image filter . By storing the intermediate filtered signal values in the storage unit , consecutive matrix multiplications can be easily performed thereby enabling complex image filtering algorithms . The storage unit can be on-chip or of f-chip . For example , the storage unit can be reali zed as random access memory, in particular SRAM or DRAM .

[0046] In addition, an electronic device is provided . The electronic device employs a camera system comprising the image sensor system . Therefore , features relating to the image sensor system are also disclosed for the electronic device , and vice vera . For example , such an electronic device may be used in the field of AR / VR, robotics , and computer vision, to name only a few . Advantages of such electronic device correspond to the advantages of the image sensor system .

[0047] In addition, a method for operating an image sensor system is provided . The method is preferably carried out using the image sensor system as described above . This means that all features disclosed for the image sensor system are also disclosed for the method for operating the image sensor system, and vice versa .

[0048] According to at least one embodiment , the method comprises generating signal values by a pixel array, wherein the pixel array comprises a plurality of pixels each comprising a photosensitive stage configured to generate the signal values .

[0049] Each pixel may generate one or more signal values . In case of more than one signal values , these signal values may belong to di f ferent integration periods . Di f ferent integration periods may belong to the same frame . Thus , the signal values can be used to reconstruct an image captured by the pixel array and / or to calculate an image velocity . In addition, each pixel may generate one or more reset levels . The reset levels may indicate noise in each pixel during the integration periods .

[0050] The method further comprises reading out and digiti ze , by a readout circuit , signal values from a subset of pixels of the pixel array .

[0051] The signal values are analog values , for example charge or voltage signals . The signal values may be digiti zed by an analog-to-digital converter of the readout circuit to generate digiti zed signal values . The subset of pixels may include all pixels of the pixel array or a group of pixels , for example one or several rows or columns of the pixel array . For example , the readout circuit reads out several rows of pixels simultaneously by means of several ADCs arranged in parallel .

[0052] The method comprises applying, by a programmable image filter comprising a memory having stored a coef ficient matrix, the coef ficient matrix onto the digiti zed signal values to generate filtered signal values , and outputting the filtered signal values .

[0053] Applying the coef ficient matrix onto the digiti zed signal values corresponds to a filter operation . The coef ficient matrix may be a predefined matrix, for example a unity matrix, a Prewitt matrix, a Sobel matrix, a Krisch matrix, a Laplace matrix, or a Gaussian matrix . Applying the coef ficient matrix can mean that a multiply-accumulate operation is performed with the coef ficient matrix and a matrix representation of the digiti zed signal values .

[0054] Advantages of the method correspond to the advantages described above in context of the image sensor system .

[0055] According to at least one embodiment of the method, generating the signal values comprises , for each pixel of the pixel array, generating a first signal value during a first integration period and storing it on a first storage element of a sample-and-hold stage of the pixel , and generating a second signal value during a second integration period and storing it on a second storage element of the sample-and-hold stage . The first integration period and the second integration period may belong to the same frame . Thus , two signal values of the same frame can be stored at pixel level , which allows to calculate a signal velocity value for each pixel .

[0056] According to at least one embodiment of the method, the method further comprises , for each pixel of the subset of pixels , determining, by a calculation circuit , a signal velocity value based on a di f ference between the first signal value and the second signal value , and on a duration of the second integration period . Advantageously, the signal velocity value can be determined at pixel / sensor level .

[0057] According to at least one embodiment of the method, the filtered signal values correspond to gradients along lateral directions of an image captured by the pixel array, and the method further comprises determining, based on the gradients and the signal velocity values , an optical flow of the image .

[0058] In this case , for example , the coef ficient matrix corresponds to a Prewitt , Sobel , Krisch and / or Laplace matrix, such that the programmable image filter is configured as gradient detector for outputting gradients along lateral directions of an image captured by the pixel array . By using the gradients and the signal velocity values , an optical flow of the image can be determined . The gradients and the signal velocity values can be calculated in parallel . This allows to simultaneously output image data, and all the data required to calculate the optical flow . The calculations of gradients , and velocity can be done at pixel / sensor level thereby eliminating the need for software postprocessing . By calculating the optical flow obj ects within the scene captured by the image sensor system can quickly identi fied and tracked .

[0059] The following description of figures may further illustrate and explain aspects of the image sensor system, the electronic device , and the method for operating the image sensor system . Components , parts and steps of the image sensor system, the electronic device , and the method, respectively, that are functionally identical or have an identical ef fect are denoted by identical reference symbols . Identical or ef fectively identical components and parts might be described only with respect to the figures where they occur first . Their description is not necessarily repeated in successive figures .

[0060] Figures 1 to 3 show exemplary embodiments of an image sensor system .

[0061] Figures 4 to 6 show further details of exemplary embodiments of an image sensor system .

[0062] Figure 7 shows further details of a programmable image filter according to an embodiment .

[0063] Figure 8 shows a pixel architecture according to an embodiment .

[0064] Figure 9 shows a timing diagram of operating a pixel according to an embodiment .

[0065] Figure 10 shows an example representation of storing states of a pixel according to an embodiment . Figure 11 shows an exemplary embodiment of an electronic device .

[0066] Figure 12 shows an exemplary embodiment of a method of operating an image sensor system .

[0067] In figure 1 an exemplary embodiment of an image sensor system 1 is shown . The image sensor system 1 comprises a pixel array 10 with a plurality of pixels 12 ( shown in detail in figure 8 ) , each of the pixels 12 comprising a photosensitive stage 17 ( shown in detail in figure 8 ) configured to generate signal values .

[0068] It further comprises a readout circuit 20 ( shown in detail in figures 4 to 6 ) . The readout circuit 20 is coupled to the pixel array 10 and being configured to read out and digiti ze the signal values from a subset of pixels 12 of the pixel array 10 . The readout circuit 20 may be seen as part of the pixel array 10 , as indicated in figure 1 .

[0069] The image sensor system 1 further comprises a programmable image filter 30 coupled to the readout circuit 20 , the programmable image filter 30 comprising a memory 32 ( shown in detail in figures 4 to 6 ) having stored a coef ficient matrix, the programmable image filter 30 being configured to apply the coef ficient matrix onto the digiti zed signal values DSV to generate and output filtered signal values FSV . In some implementations , however, the filtered signal values FSV may correspond to the original digiti zed signal values DSV . In particular, the programmable image filter 30 may be implemented as clear filter and may store a unity matrix as coef ficient matrix that leaves the digiti zed signal values unchanged upon application . As shown in figure 1, the image sensor system 1 may comprise one or more inputs for receiving instructions. For example, the programmable image filter 30 comprises an input for receiving an output selection signal OSEL. The output selection signal OSEL may be a user input to select a filter operation by selecting a coefficient matrix. In this way, the programmable image filter 30 can be programmed by selecting at least one coefficient matrix stored in the memory 32.

[0070] For example, the coefficient matrix to be selected is a unity matrix, a gradient matrix, such as Prewitt, Sobel, Krisch and / or Laplace matrices, a Gaussian matrix, an inverse of a pixel point spread function in matrix form, and / or an image sharpness matrix. The memory 32 may store a plurality of such coefficient matrices, or different coefficient matrices may be loaded to the memory 32 subsequently, based on the output selection signal OSEL. That a coefficient matrix is loaded to the memory 32 can mean that it is provided to the memory 32 by another external memory storing different coefficient matrices. Based on the selected coefficient matrix, different filter operations are performed with the digitized signal values DSV.

[0071] Further, the pixel array 10 may comprise an input for receiving image sensor settings ISET. The image sensor settings ISET may be a user input and may define whether the pixel array is used for image or video capturing, or may define an ADC bit mode and / or timing register settings, for example .

[0072] In figure 2 a modification of the image sensor system 1 according to another exemplary embodiment is shown. The embodiment according to figure 2 is different from the embodiment according to figure 1 in that the image sensor system 1 further comprises a further output for outputting image data that is not filtered . That is , the image data refers to the original digiti zed signal values DSV . The digiti zed signal values DSV and the filtered signal values FSV are output in parallel . In other words , the digiti zed signal values DSV are used as input for the programmable image filter 30 but are also bypassed to serve as parallel output of the image sensor system 1 .

[0073] In figure 3 a further modi fication of the image sensor system 1 according to another exemplary embodiment is shown . In this example , the image sensor system 1 additionally comprises a second programmable image filter 30 ' and a calculation circuit 40 .

[0074] The first and the second programmable image filter 30 , 30 ' may be implemented as gradient detectors for outputting gradients along lateral directions of an image captured by the pixel array 10 . For example , the first programmable image filter 30 determines gradients along a hori zontal direction of the image , while the second programmable image filter 30 ' determines gradients along a vertical direction of the image . However, the first and the second programmable image filter 30 , 30 ' can also be combined / merged to a single programmable image filter 30 . I f the programmable image filter ( s ) 30 is / are implemented as gradient detectors , they may have stored Prewitt , Sobel , Krisch and / or Laplace matrices in its / their memory .

[0075] The calculation circuit 40 is also coupled to the readout circuit 20 and configured to , for each pixel of the subset of pixels 12 , determine a signal velocity value SW. The signal velocity value SW is based on a di f ference between a first signal value and a second signal value , and on a duration of an integration period . The first and the second signal value may be signal values of a single pixel 12 and generated during subsequent integration periods . The first and the second signal values may have been stored temporarily in a pixel level storage element .

[0076] In the example according to figure 3 , where the programmable image filter 30 is programmed as gradient detector, the gradient detector together with the calculation circuit 40 form an optical flow engine of the image sensor system 1 . The programmable image filter 30 and the calculation circuit 40 are arranged in parallel . Thus , independent computation blocks are formed to compute gradients in x, y, and time . The optical flow engine enables to output all data required to determine an optical flow of an image captured by the image sensor system 1 . That is , the optical flow engine outputs a gradient of the image in time and gradients of the image along lateral directions x, y .

[0077] The image sensor system 1 may further comprise an output buf fer 50 , in which the output data of the programmable image filter 30 and the calculation circuit 40 may be stored temporarily and which may be configured to provide the respective output data . By means of the output buf fer 50 , the output data may be sequenced for transmission . The output buf fer 50 may comprise further computation blocks to combine the gradients and signal velocity values into optical flow data . Optionally, the image sensor system 1 further comprises a parallel output for the digiti zed signal values DSV, as indicated in figure 3 .

[0078] In figure 4 an exemplary image sensor system 1 is shown in more detail . It shows that the readout circuit 20 is arranged between the pixel array 10 and the programmable image filter 30 . The readout circuit 20 comprises an analog-to-digital converter 22 , ADC 22 , that is coupled at its input side to the pixel array 10 and at its output side to a buf fer 24 of the readout circuit 20 . The buf fer 24 is configured to temporarily store the digiti zed signal values DSV from the subset of pixels 12 and to provide them to the programmable image filter 30 . In this example , the image sensor system 1 comprises a common ADC 22 for the plurality of pixels 12 . The ADC 22 may be configured to read out the signal values of one row of the pixel array 10 . The rows of pixels 12 may be read out successively . The buf fer 24 may be implemented as SRAM buf fer, and can be on-chip or of f-chip . In case of an on-chip buf fer 24 , advantageously, less read / write controllers and / or memory addressing components are required . The buf fer 24 may be configured to store the digiti zed signal values DSV of the subset of pixels 12 , for example K rows of the pixel array 10 , K being an integer number . Each row of the pixel array 10 may comprise M columns of pixels 12 , M being an integer number . Thus , the buf fer 24 may be configured to store the digiti zed signal values DSV as a KxM matrix representation . After the digiti zed signal values DSV from the subset of pixels 12 are forwarded to the programmable image filter 30 for further processing, the readout circuit 20 may be configured to read out signal values from a further subset of pixels 12 , for example the next K rows of the pixel array 10 . As in figure 2, the digitized signal values DSV may also form a separate output of the image sensor system 1.

[0079] The pixel array 10 may comprise N columns and M rows of pixels 12, N and M being integer numbers. Further, the pixel array 10 may comprise a row driver 14 and a column driver 16, which are controlled by a control circuit 60, which may be implemented as timing and sequence generator. The control circuit 60 may also control the readout circuit 20, and the programmable image filter 30, as indicated in figure 4.

[0080] The programmable image filter 30 comprises the memory 32 in which the coefficient matrix or a plurality of coefficient matrices are stored. Which coefficient matrix is selected can be controlled by the output selection signal OSEL. This allows the user to perform e.g. gradient calculations using different types of matrices. Instead, if a unity matrix is used, then the programmable image filter 30 directly only outputs the image data. By this approach, the image sensor system 1 is highly programmable and can be used for many different applications including edge detection, etc. The programmable image filter further comprises a multiplieraccumulator unit 34, and a storage unit 36. Details of the storage unit 36 are described below in context of figure 7. The coefficient matrix can be realized as KxK matrix or kernel. Thus, the memory 32 may be configured to store KxK matrices .

[0081] In figure 5 a modification of the image sensor system 1 according to figure 4 is shown. Instead of a common ADC 22 for the pixels 12 in the pixel array 10, several ADCs 22 are used to read out a plurality of rows of the pixel array 10 simultaneously. In case of K ADCs 22, this enables the simultaneous readout of K consecutive rows. The ADCs 22 may be arranged in an ADC array. The digitized signal values DSV are then temporarily stored in respective cells of the buffer 24. Thus, the buffer may be configured to store at least K x M (number of columns) digitized signal values DSV. As in figure 2, the digitized signal values DSV may form a separate output of the image sensor system 1.

[0082] In figure 6 an image sensor system 1 according to figure 3 is shown in further detail. The pixel array 20 and the readout circuit 20 may be implemented as in figure 5. Apart from the programmable image filter 30 the image sensor system 1 further comprises a second programmable image filter 30' and a calculation circuit 40. The programmable image filter 30 and the second programmable image filter 30' can also be combined and / or merged. The programmable image filters 30, 30' can be implemented as gradient detector (s) for determining image gradients along lateral directions, such that edge data is generated. Thus, the programmable image filters 30, 30' may store gradient matrices like Prewitt, Sobel, Krisch and / or Laplace matrices in their respective memories 32, 32' . The calculation circuit 40 is configured to determine signal velocity values SW for each pixel 12. The signal velocity values SW and the filtered signal values FSV, in particular the edge data, are provided to the output buffer 50. For determining the signal velocity values SW two signal values for each pixel 12 are required. In this case, the pixel array 10 may comprise a sample-and-hold stage 18 (shown in figure 8) to temporarily store the signal values. Further, the buffer 24 may comprise additional cells to store the two signal values of each pixel 12 in digitized form. The digitized signal values DSV may form a separate output of the image sensor system 1, as illustrated in figure 3. In figure 7 an exemplary embodiment of the programmable image filter 30 is shown . The programmable image filter 30 comprises an input 37 for receiving a matrix or vector representation of the digiti zed signal values DSV . It further comprises the storage unit 36 to temporarily store the digiti zed signal values DSV . It further comprises the memory 32 having stored the coef ficient matrix, which is selectable by the output selection signal OSEL . The programmable image filter 30 further comprises a multiply-accumulate circuit 38 , 39 including a vector or matrix multiplication unit 38 and an accumulation unit 39 . The programmable image filter 30 can be operated as follows : The multiply-accumulate circuit 38 , 39 receives the digiti zed signal values DSV from the storage unit 36 and a first coef ficient matrix from the memory 32 . Then, the first coef ficient matrix is applied onto the digiti zed signal values DSV to generate intermediate filtered signal values I FSV . This can mean that the vector or matrix multiplication unit 38 and the accumulation unit 39 perform multiply-accumulate operations using a matrix representation of the digiti zed signal values DSV and the first coef ficient matrix . Then, the intermediate filtered signal values I FSV are saved in the storage unit 36 . In a next step, the multiply-accumulate circuit 38 , 39 receives the intermediate filtered signal values I FSV from the storage unit 36 and a second coef ficient matrix from the memory 32 . Then, the second coef ficient matrix is applied onto the intermediate filtered signal values I FSV to generate further intermediate or final filtered signal values FSV . Again, this can mean that the vector or matrix multiplication unit 38 and the accumulation unit 39 perform multiply-accumulate operations using a matrix representation of the intermediate filtered signal values I FSV and the second coef ficient matrix . I f further intermediate filtered signal values are generated, the process may be repeated with di f ferent coef ficient matrices until final filtered signal values are generated . The final filtered signal values FSV can then be output . For example , the first coef ficient matrix is a Gaussian smoothing matrix, and the second coef ficient matrix may be a Sobel matrix for edge detection . Successively applying coef ficient matrices onto digiti zed signal values DSV or intermediate filtered signal values I FSV may include consecutive matrix multiplications , consecutive convolutions , convolutions followed by matrix multiplications , or matrix multiplications followed by convolutions . Matrix multiplication entails the element-wise multiplication of two input matrices and the summation of the resulting products to generate a scalar value for each position of an output matrix . In contrast , convolution entails the sliding of a kernel across the input matrix, computing the element-wise product between the kernel and the overlapping sub-matrix . These products are then summed to produce a scalar value for each position of the kernel on the input matrix .

[0083] With figure 8 an exemplary pixel arrangement of the image sensor system 1 is shown . The pixel arrangement shown in figure 8 represents a single pixel 12 of the pixel array 10 . The pixel 12 comprises a photosensitive stage 17 . The photosensitive stage 17 comprises a photodiode PD that is coupled to a di f fusion node DN via a trans fer switch TX . During respective integration periods the photodiode PD accumulates charge carriers by conversion of electromagnetic radiation . At an end of the integration period the trans fer switch is activated to trans fer the charge carriers to the di f fusion node DN . The photosensitive stage may further comprise a gain capacitor Cig that is coupled to the diffusion node via a gain switch DCG. The gain capacitor and the gain switch may be referred to as double conversion gain stage. By activating the gain switch DCG the capacitance of the diffusion node DN is increased by the capacitance of the gain capacitor Cig, resulting in a reduced gain for the charge carriers. Thus, the dynamic range of the pixel 12 can be increased. The photosensitive stage 14 further comprises a reset switch RST that is coupled between the diffusion node DN and a pixel supply voltage VDD. By activating the reset switch RST (and, if applicable, the gain switch DCG) , the diffusion node DN is connected to the pixel supply voltage VDD and thus reset.

[0084] The pixel 12 of figure 8 further shows a sample-and-hold stage 18, S / H stage 18, which is optional in some implementations. The S / H stage 18 is coupled to the diffusion node via a first source follower SF1. The S / H stage 18 comprises a first capacitor bank and a second capacitor bank that are arranged in parallel. The first capacitor bank comprises a first capacitor (denoted C3) as first storage element connected in cascade with a second capacitor (denoted Cl) as second storage element. The second capacitor bank comprises a third capacitor (denoted C4) as third storage element connected in cascade with a fourth capacitor (denoted C2) as fourth storage element. The first to fourth capacitors are controllable via a first to fourth switch (denoted SI to S4) , respectively. The first storage element is configured to store a first signal value generated during a first integration period and the second storage element is configured to store a second signal value generated during a second integration period. The third storage element is configured to store a first reset level generated during the first integration period and the fourth storage element is configured to store a second reset level generated during a second integration period.

[0085] The S / H stage 18 shown in figure 8 further comprises a precharge transistor PC that may be configured to reset the storage elements by applying a negative pixel supply voltage VSS and / or to drive the first source follower SF1. Further, each capacitor bank is coupled at its output side to a respective second source follower SF2, SF2' . An output of the second source follower SF2, SF2' is coupled to a bus via a respective select switch SEL, SEL' . Thus, different pixel output signals V_OUT1, V_OUT2 may be transferred via the bus to the readout circuit 20.

[0086] In figure 9 a possible timing diagram for operating the pixel 12 according to figure 8 is shown. At a time to the transfer switch TX, the gain switch DCG, and the reset switch RST are activated to reset the diffusion node and to start a new frame of image capturing. The period between time to and a time ti defines a first integration period in which the photodiode PD accumulates charge carriers. Before time ti, the switches S2 and S4 are activated to transfer a first reset level to the capacitor C4. At time ti, the transfer switch is activated to transfer the accumulated charge carriers to the diffusion node DN. Immediately after this pulse, the switches SI and S3 are activated to transfer these charge carriers to the capacitor C3, where they are stored as first signal value. Then, the diffusion node DN is reset by activating the gain switch DCG and the reset switch RST. The period between time ti and a time to defines a second integration period in which the photodiode PD accumulates charge carriers. Before time to, the switch S2 is activated to transfer a second reset level of the pixel 12 to the capacitor C2 . At time t2 , the trans fer switch is activated to trans fer the charge carriers accumulated in the second integration period to the di f fusion node DN . Immediately after this pulse , the switch S I is activated to trans fer these charge carriers to the capacitor Cl , where they are stored as second signal value . Then, the di f fusion node DN is reset by activating the gain switch DCG and the reset switch RST and the process restarts in the next frame .

[0087] It is noted that the di f fusion node DN is not reset between the trans fers of the first reset level and the first signal value ( charge carriers accumulated in the first integration period) , and between the trans fers of the second reset level and the second signal value ( charge carriers accumulated in the second integration period) . Thus , the respective reset levels indicate noise that is correlated with the noise of the respective signal values . Therefore , correlated double sampling can be performed to correct the signal values . This is illustrated in figure 10 showing an example representation of storing states in the sample-and-hold stage 18 , in particular indicating the amount of charges stored in the respective storing elements Cl to C4 . The first reset level stored on capacitor C4 corresponds to noise in the signal stored on capacitor C3 . Accordingly, the second reset level stored on capacitor C2 corresponds to noise in the signal stored on capacitor Cl . Further, since the capacitors Cl and C3 are arranged cascaded and the charges are distributed to the two capacitors during the first charge trans fer, the first signal value stored on capacitor C3 is also contained in the second signal value stored on capacitor Cl . Given the above explanations , the following calculations can be performed by the calculation circuit 40 processing the digiti zed signal values .

[0088] The first signal value can be accessed with correlated double sampling by considering the first reset level . Thus , a corrected first signal value can be obtained . The corrected first signal value is denoted s i in the following . The second signal value can be accessed with correlated double sampling by considering the second reset level . Thus , a corrected second signal value can be obtained . The corrected second signal value is denoted s2 in the following . Then, a signal velocity value SW can be determined by :

[0089] The plurality of signal velocity values SW form a gradient of the image in time , dl / dt . As mentioned above , the programmable image filter 30 , implemented as gradient detector, may determine gradients of the image along lateral directions , dl / dx and dl / dy . An optical flow can than determined by the following optical flow equation : where u = dx / dt and v = dy / dt .

[0090] Referring now to Figure 11 , an electronic device 100 is shown which includes a camera system 101 comprising an image sensor system 1 according to one of the implementations described in this disclosure . Such electronic device 100 may speci fically be adapted for AR, VR, robotics and / or computer vision applications . Figure 12 shows a block diagram of an example implementation of a method for operating an image sensor system 1 . For example , such image sensor system 1 is implemented according to one of the examples described above . The method comprises the following steps that are not necessarily carried out in this order but can be carried out in this order .

[0091] In a step S 10 , signal values are generated by a pixel array, wherein the pixel array having a plurality of pixels each comprising a photosensitive stage configured to generate the signal values . Generating the signal values may comprise , for each pixel of the pixel array, generating a first signal value during a first integration period and storing it on a first storage element of a sample-and-hold stage of the pixel , and generating a second signal value during a second integration period and storing it on a second storage element of the sample-and-hold stage .

[0092] In a step S20 , signal values from a subset of pixels of the pixel array, are read out and digiti zed, by a readout circuit .

[0093] In a step S30 , a programmable image filter comprising a memory having stored a coef ficient matrix, applies the coef ficient matrix onto the digiti zed signal values to generate filtered signal values .

[0094] In an optional step S31 , a further programmable image filter comprising a memory having stored a further coef ficient matrix, applies the further coef ficient matrix onto the digiti zed signal values to generate further filtered signal values , and outputs the further filtered signal values . For example , the programmable image filter and the further programmable image filter are reali zed as gradient detectors for detecting gradients along lateral directions of the image . For example , the programmable image filter filters the image data to detect edges along the hori zontal direction of the image , and the further programmable image filter filters the image data to detect edges along the vertical direction of the image , or vice vera . The programmable image filters can also be combined to form a single programmable image filter .

[0095] In an optional step S32 , for each pixel of the subset of pixels , a signal velocity value is determined, by a calculation circuit , based on a di f ference between the first signal value and the second signal value , and on a duration of the second integration period .

[0096] In a step S40 , the programmable image filter outputs the filtered signal values . Further filtered signal values and the signal velocity values , as described in context of step S31 and step 32 , may also be output .

[0097] In an optional step S50 , i f the filtered signal values correspond to gradients along lateral directions of an image captured by the pixel array and i f signal velocity values are calculated, an optical flow of the image is determined, based on the gradients and the signal velocity values .

[0098] After step S40 or S50 , the method may start over with a new set of signal values generated by the pixel array .

[0099] The embodiments of the improved imaging concept disclosed herein have been discussed for the purpose of familiari zing the reader with novel aspects of the implementation of the improved imaging concept . Although preferred embodiments have been shown and described, many changes , modi fications , equivalents , and substitutions of the disclosed concepts may be made by one having skill in the art without departing from the scope of the claims .

[0100] In particular, the implementation of the improved imaging concept is not limited to the disclosed embodiments , and gives examples of many alternatives possible for the features included in the embodiments discussed . However, it is intended that any modi fications , equivalents , and substitutions of the disclosed concepts be included within the scope of the claims which are appended hereto .

[0101] Features recited in separate dependent claims may be advantageously combined . Moreover, reference signs used in the claims are not limited to be construed as limiting the scope of the claims .

[0102] Furthermore , as used herein, the term "comprising" does not exclude other elements . In addition, as used herein, the article "a" is intended to include one or more than one component or element , and is not limited to be construed as meaning only one .

[0103] This patent application claims priority from US patent application 63 / 635 , 319 , the disclosure content of which is hereby included by reference . References

[0104] 1 image sensor system

[0105] 10 pixel array

[0106] 12 pixel

[0107] 14 row driver

[0108] 16 column driver

[0109] 17 photosensitive stage

[0110] 18 sample-and-hold stage

[0111] 20 readout circuit

[0112] 22 analog-to-digital converter

[0113] 24 buf fer

[0114] 30 , 30 ' programmable image filter

[0115] 32 , 32 ' memory

[0116] 34 multiplier-accumulator unit

[0117] 36 storage unit

[0118] 37 input

[0119] 38 vector or matrix multiplication unit

[0120] 39 accumulation unit

[0121] 40 calculation circuit

[0122] 50 output buf fer

[0123] 60 control circuit

[0124] 100 electronic device

[0125] 101 camera system

[0126] Cig gain capacitor

[0127] DCG gain switch

[0128] DN di f fusion node

[0129] DSV digiti zed signal value

[0130] FSV filtered signal value

[0131] I FSV intermediate filtered signal value

[0132] ISET image sensor setting

[0133] OSEL output selection signal

[0134] PC precharge transistor PD photodiode

[0135] RST reset switch

[0136] S1-S4 switch

[0137] S10-S50 step SEL, SEL' select switch

[0138] SEI, SF2, SF2' source follower

[0139] SW signal velocity value to- t2 time

[0140] TX transfer switch VDD, VSS pixel supply voltage

[0141] V OUT pixel output data

Claims

Claims1. An image sensor system (1) , comprising:- a pixel array (10) with a plurality of pixels (12) , each of the pixels (12) comprising a photosensitive stage (14) configured to generate signal values;- a readout circuit (20) coupled to the pixel array (10) and being configured to read out and digitize the signal values from a subset of pixels (12) of the pixel array (10) ;- a programmable image filter (30) coupled to the readout circuit (20) , the programmable image filter (30) comprising a memory (32) having stored a coefficient matrix, the programmable image filter (30) being configured to apply the coefficient matrix onto the digitized signal values (DSV) to generate and output filtered signal values (FSV) .

2. The image sensor system (1) according to claim 1, wherein the programmable image filter (30) is configured to be programmed by selecting at least one coefficient matrix stored in the memory (32) .

3. The image sensor system (1) according to one of claims 1 to 2, wherein the memory (32) is configured to- store a unity matrix, such that the programmable image filter (30) can be programmed as clear filter for outputting pure image data of an image captured by the pixel array (10) ; and / or- store a gradient matrix, in particular Prewitt, Sobel, Krisch and / or Laplace matrices, such that the programmable image filter (30) can be programmed as gradient detectorfor outputting gradients along lateral directions of an image captured by the pixel array (10) .

4. The image sensor system (1) according to one of claims 1 to 3, wherein the memory (32) is further configured to- store a Gaussian matrix, such that the programmable image filter (30) can be programmed as Gaussian image filter; and / or- store an inverse of a pixel point spread function in matrix form, such that the programmable image filter (30) can be programmed to function as a filter that compensates the optical crosstalk between pixels; and / or- store an image sharpness matrix, such that the programmable image filter (30) can be programmed as an image sharpness improvement filter.

5. The image sensor system (1) according to one of claims 1 to 4, wherein each of the pixels (12) further comprises a sample-and-hold stage (18) configured to temporarily store the signal values.

6. The image sensor system (1) according to claim 5, wherein the sample-and-hold stage (18) of each pixel (12) comprises a first storage element configured to store a first signal value generated during a first integration period and a second storage element configured to store a second signal value generated during a second integration period.

7. The image sensor system (1) according to one of claims 5 to 6, wherein the sample-and-hold stage (30) of each pixel (12) further comprises a third storage element configured to store a first reset level generated during a first integration period and a fourth storage element configured tostore a second reset level generated during a second integration period.

8. The image sensor system (1) according to one of claims 5 to 7, wherein the sample-and-hold stage (18) of each pixel (12) comprises a first capacitor bank and a second capacitor bank arranged in parallel, wherein the first capacitor bank comprises a first capacitor (C3) as first storage element connected in cascade with a second capacitor (Cl) as second storage element, and wherein the second capacitor bank comprises a third capacitor (C4) as third storage element connected in cascade with a fourth capacitor (C2) as fourth storage element.

9. The image sensor system (1) according to one of claims 1 to 8, wherein the image sensor system (1) is configured to output the digitized signal values (DSV) and the filtered signal values (FSV) in parallel.

10. Image sensor system (1) according to claim 6, wherein the image sensor system (1) further comprises a calculation circuit (40) coupled to the readout circuit (20) and configured to, for each pixel (12) of the subset of pixels (12) , determine a signal velocity value (SW) based on a difference between the first signal value and the second signal value, and on a duration of the second integration period .

11. The image sensor system (1) according to claim 10, wherein the programmable image filter (30) is programmed as gradient detector for an image captured by the pixel array (10) , and wherein the gradient detector together with thecalculation circuit (40) form an optical flow engine integrated into the image sensor system (1) .

12. The image sensor system (1) according to claim 10 or 11, wherein the programmable image filter (30) and the calculation circuit (40) are arranged in parallel.

13. The image sensor system (1) according to one of claims 1 to 12, wherein the readout circuit (20) comprises at least one analog-to-digital converter (22) coupled at its input side to the pixel array (10) and at its output side to a buffer (24) of the readout circuit (20) , the buffer (24) being configured to temporarily store the digitized signal values (DSV) from the subset of pixels (12) and to provide them to the programmable image filter (30) .

14. The image sensor system (1) according to one of claims 1 to 13, wherein applying the coefficient matrix comprises a multiply-accumulate operation with the coefficient matrix and a matrix representation of the digitized signal values (DSV) .

15. The image sensor system (1) according to one of claims 1 to 14, wherein the programmable image filter (30) further comprises a storage unit (36) and a multiply-accumulate circuit (38, 39) , wherein the multiply-accumulate circuit (38, 39) is configured to- receive the digitized signal values (DSV) from the storage unit (36) and a first coefficient matrix from the memory (32) ,- apply the first coefficient matrix onto the digitized signal values (DSV) to generate intermediate filtered signal values (IFSV) ,- store the intermediate filtered signal values (IFSV) in the storage unit (36) ,- receive the intermediate filtered signal values (IFSV) from the storage unit (36) and a second coefficient matrix from the memory (32) , and- apply the second coefficient matrix onto the intermediate filtered signal values (IFSV) to generate further intermediate or final filtered signal values (FSV) .

16. An electronic device (100) with a camera system (101) comprising the image sensor system (1) according to one of claims 1 to 15.

17. A method for operating an image sensor system, the method comprising :- generating (S10) signal values by a pixel array, wherein the pixel array having a plurality of pixels each comprising a photosensitive stage configured to generate the signal values;- reading out and digitize (S20) , by a readout circuit, signal values from a subset of pixels of the pixel array;- applying (S30) , by a programmable image filter comprising a memory having stored a coefficient matrix, the coefficient matrix onto the digitized signal values to generate filtered signal values, and- outputting (S40) the filtered signal values.

18. The method according to claim 17, wherein generating (S10) the signal values comprises, for each pixel of the pixel array, generating a first signal value during a first integration period and storing it on a first storage element of a sample-and-hold stage of the pixel, and generating asecond signal value during a second integration period and storing it on a second storage element of the sample-and-hold stage .

19. The method according to claim 18, further comprising, for each pixel of the subset of pixels, determining (S32) , by a calculation circuit, a signal velocity value based on a difference between the first signal value and the second signal value, and on a duration of the second integration period.

20. The method according to claim 19, wherein the filtered signal values correspond to gradients along lateral directions of an image captured by the pixel array, and the method further comprises determining (S50) , based on the gradients and the signal velocity values, an optical flow of the image .

Images