Method, image sensor and color space conversion unit for color imaging using arbitrary color filter array event data

Abstract

The invention relates to a method, an image sensor and a color space conversion unit for color imaging using arbitrary color filter array event data. An image sensor for color imaging using arbitrary color filter array event data is provided. The image sensor comprises a plurality of color imaging pixels and a plurality of color event pixels. An image signal of a first color imaging pixel included in the plurality of color imaging pixels is determined based on a first color signal of the first color imaging pixel and at least color event data of one or more color event pixels included in the plurality of color event pixels. The color event data is generated in a manner having a temporal relationship with generation of the first color signal.

Description

Technical Field

[0001] The present disclosure relates to an image sensor, and more particularly, to an image sensor for color imaging using arbitrary color filter array event data. Background Art

[0002] The related art of patent specifications for color imaging methods relates to the combination of a complementary metal oxide semiconductor (CMOS) image sensor (CIS) and an event-based vision sensor (EVS). CIS is widely used in digital cameras and other imaging devices due to its power efficiency and fast readout speed. Although CIS provides great image and / or video capture capabilities, one of its limitations is that it is difficult for a conventional image / video sensor to provide ultra-high frame rate and ultra-high speed capture capabilities, which can be useful in various applications such as high-speed motion, machine vision, gaming, and artificial intelligence sensing fields. In typical operation, CIS utilizes active pixel sensing elements that require a specific exposure time interval to integrate small photocurrents and then output the image data in the acquisition order in an image frame. To capture high-speed motion, the active pixel sensing elements must operate at a very high frame rate. This results in a large amount of data being output by the conventional active pixel sensing elements. This output data typically contains a very high level of redundancy between frames, most of which can be used to convey the same static or slowly changing background of the field of view. In other words, a large amount of background information is continuously sampled, resampled, output, and then reprocessed using conventional active pixel sensing elements. Attempts to provide such ultra-high frame rate and ultra-high speed capabilities for a typical image / video sensor have led to compromise solutions that provide poorer quality image / video capture compared to their normal image sensor counterparts.

[0003] On the other hand, EVS (also known as a neuromorphic sensor) is the latest advancement in simulating the functions of the human eye and brain. These sensors detect changes in a scene and only capture relevant information, resulting in low power consumption and high dynamic range. Summary of the Invention

[0004] One aspect of the present disclosure provides an image sensor. The image sensor includes a plurality of color imaging pixels and a plurality of color event pixels. The image signal of a first color imaging pixel among the plurality of color imaging pixels is determined based on the color signal of a first color image pixel and the color event data of one or more color event pixels among the plurality of color event pixels. The color event data is detected or otherwise received in a manner having a temporal relationship with the generation of the color signal.

[0005] In some embodiments, the color event data is generated during the exposure or integration time interval of the first color imaging pixel. The color event data may be generated within the same event detection time interval as the exposure or integration time interval of the first color imaging pixel. In some embodiments, the color event data is received within a specific time interval after the time of receiving the color signal. In some embodiments, the color event data is generated within a specific time interval before the time of generating the color signal. In some embodiments, a first set of color event data is received within a first time interval before the time of generating the color signal, and a second set of color event data is received within a second time interval after the time of generating the color signal. In such embodiments, the image signal of the first color imaging pixel among the plurality of color imaging pixels is determined based on the color signal of the first color image pixel, the first color event data of one or more color event pixels among the plurality of color event pixels, and the second color event data of one or more color event pixels among the plurality of color event pixels.

[0006] Another aspect of the present disclosure provides a color imaging method. The method includes: defining a target CIS color channel; selecting one or more EVS pixels depending on the proximity to the target CIS color channel; applying a set of color conversion model parameters to the EVS data of the one or more EVS pixels to generate color space data; and fusing the color space data with the image signal of the target CIS color channel.

[0007] Another aspect of the present disclosure provides an EVS to CIS color space conversion unit. The EVS to CIS color space conversion unit performs the following steps: defining a target CIS color channel; selecting one or more EVS pixels depending on the proximity to the target CIS color channel; applying a set of color conversion model parameters to the EVS data of the one or more EVS pixels to generate color space data; and returning the generated color space data. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Advantages of the present disclosure will be best understood from the following detailed description when read in conjunction with the accompanying Figure 1 drawings. It should be noted that, in accordance with standard practice in the industry, the various features are not drawn to scale. In fact, the dimensions of the various features may be arbitrarily increased or reduced for clarity of discussion.

[0009] Figure 1 Illustrates a hybrid image sensor according to some embodiments of the present disclosure.

[0010] Figure 2 Illustrates a flowchart of a color imaging method according to some embodiments of the present disclosure.

[0011] Figure 3 A time diagram illustrating the logarithmic pixel illuminance and event stream according to some embodiments of the present disclosure.

[0012] Figure 4 A time diagram illustrating the logarithmic pixel illuminance and event stream according to some embodiments of the present disclosure.

[0013] Figure 5 A flowchart of a method for converting color event data into color space data according to some embodiments of the present disclosure.

[0014] Figure 6 A flowchart illustrating an error minimization process according to some embodiments of the present disclosure.

[0015] Figure 7 A schematic diagram illustrating an error minimization process according to some embodiments of the present disclosure.

[0016] Figure 8 A system diagram of an image system according to some embodiments of the present disclosure. Detailed Description

[0017] The following disclosure provides many different embodiments or examples for implementing different features of the provided subject matter. Specific examples of components and arrangements are described below to simplify the present disclosure. Of course, these are only examples and are not intended to be limiting. For example, in the following description, forming a first feature above or on a second feature may include embodiments in which the first and second features are formed in direct contact and may also include embodiments in which additional features may be formed between the first and second features such that the first and second features may not be in direct contact. Additionally, the present disclosure may repeat reference numerals and / or letters in various examples. This repetition is for simplicity and clarity purposes and does not inherently indicate a relationship between the various embodiments and / or configurations discussed.

[0018] As used herein, although terms such as "first," "second," and "third" describe various elements, components, regions, layers, and / or sections, these elements, components, regions, layers, and / or sections should not be limited by these terms. These terms may be used only to distinguish one element, component, region, layer, or section from another. Unless the context clearly indicates otherwise, terms such as "first," "second," and "third" do not imply an order or sequence when used herein.

[0019] While numerical ranges and parameters that describe the broad scope of this disclosure are approximations, the numerical values set forth in the specific examples are reported as precisely as possible. Any numerical value, however, inherently contains certain errors necessarily resulting from the standard deviation found in the respective testing measurements. Also, as used herein, the terms "substantially," "about," and "approximate" generally mean within the value or range that would be considered acceptable to a person of ordinary skill in the art. Alternatively, the terms "substantially," "about," and "approximate" mean within the acceptable standard error of the mean when considered by a person of ordinary skill in the art. The acceptable standard error can be understood by a person of ordinary skill in the art to vary depending on the technology. Except in the operating / working examples, or unless otherwise expressly specified, all numerical ranges, amounts, values, and percentages such as those for amounts of materials, duration of time, temperatures, operating conditions, ratios of amounts, and the like disclosed herein should be understood to be modified in all instances by the term "substantially," "about," or "approximate." Accordingly, unless indicated to the contrary, the numerical parameters set forth in the present disclosure and attached claims are approximations that may vary as desired. At the very least, each numerical parameter should be construed in light of the number of reported significant digits and by applying ordinary rounding techniques. Ranges may be expressed herein as from one endpoint to another endpoint or between two endpoints. All ranges disclosed herein are inclusive of the endpoints unless otherwise specified.

[0020] A frame camera equipped with a CMOS image sensor (CIS) offers numerous advantages, including synchronized images, spatially dense information, adjustable exposure, and absolute image intensity. For global shutter CIS, synchronized image capture is allowed to ensure that all pixels are exposed simultaneously. This feature eliminates the risk of motion blur or distortion that may occur during consecutive image captures. Thus, the frame camera can accurately capture fast-moving objects or scenes with a high dynamic range. The CIS provides spatially dense information, meaning that it can capture a large number of pixels in a given area. This high pixel density enables the camera to capture fine details and produce high-resolution images. Whether for scientific research, surveillance, or professional photography, a frame camera with CIS can deliver clear and detailed images.

[0021] The CIS can further incorporate features with adjustable exposure times. This feature allows the camera to adapt to different lighting conditions and capture images with optimal brightness and contrast. By adjusting the exposure settings, users can ensure that their images are correctly exposed, even in challenging lighting situations. Additionally, the CMOS image sensor provides absolute image intensity, which refers to the ability to accurately measure the light intensity in an image. This feature is particularly useful in scientific applications that require precise measurements. Using a CMOS image sensor (CIS), a frame camera can provide accurate and reliable intensity measurements, making it suitable for a variety of scientific experiments and research. The CIS is also well-suited for capturing static scenes. It is adept at capturing static images with minimal noise and distortion. This makes the CIS highly suitable for applications such as landscape photography, architectural photography, or any situation that requires a stable and clear image.

[0022] Event cameras with event-based vision sensors have revolutionized the way we capture and process visual information. Unlike traditional cameras that capture images at a fixed rate, event cameras operate on a completely different principle, offering several advantages that make them highly desirable in a variety of applications. One of the main advantages of event cameras is their ability to capture asynchronous data. Event cameras only capture and transmit data when there is a change in the scene (a change in light intensity), rather than capturing frames at a fixed rate. This means that they are extremely efficient in terms of data transmission and storage, as they only capture and transmit relevant information. This asynchronous nature allows event cameras to capture fast-moving objects with high accuracy and minimal motion blur, making the event cameras well-suited for applications such as robotics, autonomous vehicles, and sports analysis.

[0023] Another significant advantage of event cameras is their ability to provide temporally dense information. Conventional cameras capture a series of frames at a fixed rate, which can result in important details being missed between frames. In contrast, event cameras capture every single change in a scene, providing a continuous stream of information with microsecond temporal resolution. This enables event cameras to capture fast and subtle movements that conventional cameras would miss, making the event cameras suitable for applications such as object tracking, gesture recognition, and motion analysis. Event cameras are also good at capturing scenes with high dynamic range. Conventional cameras have difficulty capturing scenes with large variations in lighting conditions, often resulting in overexposed or underexposed regions. On the other hand, event cameras have a high dynamic range, allowing them to capture details in both bright and dark regions simultaneously. This makes event cameras well-suited for applications such as surveillance, outdoor imaging, and HDR imaging. Additionally, event cameras offer the advantage of low power consumption. Since event cameras only capture and transmit data when there is a change in the scene, they require significantly less power compared to conventional cameras that continuously capture frames. This makes event cameras suitable for battery-powered devices and applications where power efficiency is crucial. In summary, event cameras or event-based vision sensors are a breakthrough technology that offers several advantages over conventional cameras. The ability of the event cameras or event-based vision sensors to capture asynchronous images, provide temporally dense information, eliminate image blur, and offer high dynamic range makes them highly desirable in various fields such as robotics, autonomous vehicles, surveillance, etc. Leveraging their unique capabilities, event cameras are poised to revolutionize the way we capture and process visual information in the future.

[0024] Event-based vision sensors (EVS) are typically monochromatic sensors. Monochromatic sensors cannot capture color information from a scene. Thus, in some cases, very different colors may produce the same response on an event-based vision sensor. To overcome this problem, a color filter array can be used to capture color information.

[0025] To combine the advantages of both CIS and EVS and compensate for each other's drawbacks, methods for combining any color filter array (CFA) color-sensitive conventional (CMOS) and neuromorphic / EVS data are disclosed. The combination of CIS and EVS offers several advantages. For example, it can enable high-speed video reconstruction. CIS captures frames at a fixed rate. However, EVS only captures changes in the scene, resulting in a sparse representation of visual information. By combining the two, it may be possible to reconstruct high-speed video by using EVS data to fill the gaps between CIS frames. This allows for the capture of fast-moving objects and actions that would otherwise be missed by individual conventional CIS.

[0026] Another advantage is reduced motion blur. When a fast-moving object is captured within the integration time interval of each image frame, the CIS frame can suffer from motion blur, where the position of the fast-moving object changes between the start and end of the integration time. On the other hand, EVS captures events with high temporal resolution, resulting in less motion blur. By combining the two sensors, it is possible to reduce motion blur in the final image or video, resulting in a clearer and more detailed visual effect.

[0027] In addition, the combination of CIS and EVS data can allow for high dynamic range (HDR) imaging without ghosting. High dynamic range (HDR) imaging involves capturing multiple exposures of a scene to accurately capture both bright and dark regions. However, when an object moves between exposures, traditional HDR techniques can result in ghosting artifacts. The EVS, with its high temporal resolution, can capture events without any motion blur, allowing for frame deblurring and temporal alignment, which will result in accurate HDR imaging without ghosting artifacts.

[0028] Furthermore, the combination of these sensors makes object identification and tracking easier through color cues. The CMOS image sensor captures color information that can be used for object identification and tracking. By combining the color information captured by the CMOS image sensor with the high temporal resolution captured by the event-based vision sensor, it is possible to track objects more accurately and efficiently. The color cue provides additional information that can help distinguish an object from the background and improve the accuracy of the object identification algorithm.

[0029] The method for combining CIS and EVS is compatible with applications having multiple cameras or applications having hybrid systems. For example, the data of an EVS camera can be combined with the data of a CIS camera to output combined image data. Additionally, CIS pixels and EVS pixels can be integrated on the same sensor (e.g., on a single chip) to form a hybrid image sensor. The CIS pixels and EVS pixels can be arranged in different patterns for the hybrid image sensor according to the requirements of intensity and events. The ratio of CIS pixels to EVS pixels on the hybrid image sensor can also vary according to the requirements of intensity and events.

[0030] In some embodiments of the present disclosure, multiple event-sensing pixels and multiple image-sensing pixels can be formed together into a hybrid structure, thereby simplifying the installation of the image-sensing device.

[0031] A hybrid image sensor that combines EVS pixels and CIS pixels offers a set of advantages that make it highly desirable in the field of computer vision. The hybrid image sensor's ability to capture spatially and temporally dense images, eliminate motion blur, and provide high dynamic range imaging without ghosting makes it well-suited for a wide range of applications, including robotics, autonomous vehicles, and sports analysis. The hybrid image sensor can also provide easier object identification and tracking through a colorimeter.

[0032] To combine CIS and EVS information, one cannot rely on monochromatic event data because the monochromatic event data cannot provide any information about the chromaticity of the object that triggered the event. Applications such as slow-motion video generation or deblurring require color events to achieve satisfactory results. Therefore, a color filter array (CFA) overlaid on the EVS pixels is needed. Additionally, to maximize the sensitivity of the EVS data, a CFA different from the CFA used for CIS data (with a QE curve with a wider spectrum) can be used. Since the color imaging data and the color event data can belong to different color spaces (with different CFA patterns), some data should be converted to a common color space before being combined.

[0033] Figure 1 Illustrates a hybrid image sensor 10 according to some embodiments of the present disclosure.

[0034] As Figure 1 shown, the hybrid image sensor 10 includes a plurality of color imaging pixels 101, 102, 103, 104, 105 and a plurality of color event pixels 111, 112, 113, 114. In an embodiment, the plurality of color imaging pixels 101, 102, 103, 104, 105 and the plurality of color event pixels 111, 112, 113, 114 may be arranged to form a pixel array. The pixel array can be a two-dimensional (2D) array, where the plurality of color imaging pixels 101, 102, 103, 104, 105 and the plurality of color event pixels 111, 112, 113, 114 are arranged in a number of rows and a number of columns. Although only five color imaging pixels 101, 102, 103, 104, 105 of the hybrid image sensor 10 are labeled with reference numerals, all pixels shown in the hybrid image sensor 10 without a dashed background are color imaging pixels. In some embodiments of the present disclosure, for simplicity, the color imaging pixels are also referred to as CIS pixels. As for the pixels 111, 112, 113, 114 in the hybrid image sensor 10 with a dashed background, they are color event pixels. In some embodiments of the present disclosure, for simplicity, the color event pixels are also referred to as EVS pixels.

[0035] While in the illustrated embodiments, the plurality of CIS and EVS pixels in the hybrid image sensor 10 are arranged to have a color pattern or an RGBW CFA pattern consisting of "red", "green", "blue", and "white", in other embodiments of the present disclosure, various CFA patterns can be used for the plurality of CIS and EVS pixels in the hybrid image sensor 10, such as red, green, green, blue (RGGB), red, clear, clear, blue (RCCB), or red, blue, green, clear (RGBC), or red, green, blue, infrared (RGBIR), or cyan, yellow, yellow, magenta (CYYM), etc. In some embodiments of the present disclosure, the color imaging pixels and the color event pixels may have different CFA patterns. For example, the color imaging pixels may be arranged to have an RGBC pattern, while the color event pixels may be arranged to have a cyan, yellow, yellow, magenta (CYYM) pattern. In another example, the color imaging pixels may be arranged to have a Bayer pattern, while the color event pixels may be arranged to have a red, blue, green, clear (RBGC) pattern or a cyan, yellow, yellow, magenta (CYYM) pattern.

[0036] In some embodiments of the present disclosure, the image signal of the first color imaging pixel 101 included in the plurality of color imaging pixels 101, 102, 103, 104, 105 is determined based on the color signal of the first color image pixel 101 and the color event data of the color event pixels 111, 113, 114 included in the plurality of color event pixels 111, 112, 113, 114.

[0037] In an embodiment, the color event data of one or more color event pixels 111, 113, 114 included in the plurality of color event pixels 111, 112, 113, 114 is detected or otherwise received in a manner that has a temporal relationship with the generation of the color signal. For example, the color event data may be captured within a specific time interval after the time when the corresponding color signal is received. For another example, the color event data may be captured within a specific time interval before the time when the corresponding color signal is received.

[0038] In some embodiments, color event data is captured or otherwise generated during the exposure or integration time interval of the first color imaging pixel 101. The color event data can be generated within the same event detection time interval as the exposure or integration time interval of the first color imaging pixel. In some embodiments, the color event data is received within a specific time interval after the time at which the color signal is received. In some embodiments, the color event data is generated within a specific time interval before the time at which the color signal is generated. The specific time interval can be configured based at least on the exposure (or integration) time interval of the first color imaging pixel 101 and the readout time of the color signal.

[0039] In some other embodiments, a first set of color event data can be received within a first time interval before the time at which the color signal is generated, and a second set of color event data can be received within a second time interval after the time at which the color signal is generated. In such embodiments, the image signal of the first color imaging pixel (e.g., the first color imaging pixel 101) among the plurality of color imaging pixels is determined based on the color signal of the first color image pixel 101, the first color event data of one or more color event pixels among the plurality of color event pixels, and the second color event data of one or more color event pixels among the plurality of color event pixels. The first time interval and the second time interval can be configured in view of the exposure or integration interval of the first color imaging pixel 101.

[0040] In some embodiments of the present disclosure, one or more of the color event pixels 111, 113, 114 are color event pixels among the plurality of color event pixels 111, 112, 113, 114 that are related to the first color image pixel 101 (e.g., spatially or temporally related to the first color image pixel 101).

[0041] In some embodiments of the present disclosure, one or more of the color event pixels 111, 113, 114 are color event pixels among the plurality of color event pixels 111, 112, 113, 114 that are located closely proximate to the first color imaging pixel 101. In some embodiments of the present disclosure, the color sensed by the first color image pixel 101 is different from the color associated with one of the one or more color event pixels 111, 113, 114. However, in some embodiments of the present disclosure, the color sensed by the first color image pixel 101 can be the same as the color of one of the one or more color event pixels 111, 113, 114 used to determine the image signal of the first color imaging pixel 101.

[0042] In some embodiments of the present disclosure, the image signal of each color imaging pixel included in the plurality of color imaging pixels in the hybrid image sensor 10 is determined based on the color signal of the color image pixel and the color event data of one or more color event pixels included in the plurality of color event pixels 111, 112, 113, 114. In some embodiments of the present disclosure, one or more color event pixels are associated with each color imaging pixel in the plurality of color imaging pixels. In some embodiments of the present disclosure, the first color imaging pixel 101 is located within the region of interest of the plurality of color image pixels 101, 102, 103, 104, 105.

[0043] Figure 2 Flowchart 20 illustrating a color imaging method according to some embodiments of the present disclosure. Flowchart 20 may be executed by a processor included in the hybrid image sensor.

[0044] The method may start from block 200. In some embodiments of the present disclosure, in block 201, color CIS data is captured from the plurality of color imaging pixels 101, 102, 103, 104, 105. In block 203, the color CIS data is stored in the CIS data buffer. The CIS data buffer may be included in a memory on the hybrid image sensor. Similarly, in some embodiments of the present disclosure, in block 202, color EVS data is captured from the plurality of color event pixels 111, 112, 113, 114. In block 204, the color EVS data is stored in the EVS buffer. The EVS data buffer may be included in a memory on the hybrid image sensor. In an embodiment, block 202 may be implemented after block 201.

[0045] To combine the color CIS data and the color EVS data, in step 206, the color EVS data stored in the EVS buffer is converted to the CIS color space. In block 210, fusion of the color CIS data and the color EVS data is performed. The method ends at block 211. In some embodiments of the present disclosure, it is performed in real time Figure 2 of the method.

[0046] Figure 3 Time graph illustrating the logarithmic pixel illuminance and event stream according to some embodiments of the present disclosure.

[0047] Figure 3 Shows the EVS to CIS color space conversion, which illustrates how color events from the color event pixels 111, 112, 113, 114 are used to determine a given CIS color channel. In some embodiments of the present disclosure, events are accumulated by event time integration as shown below:

[0048]

[0049] , where p i ∈P, c i ∈C. P is a set of CIS color channels, and C is a set of EVS color channels. In some embodiments of the present disclosure, when there are four CIS color channels and four EVS color channels.

[0050] P = {p1, p2, p3, p4} (2)

[0051] C = {c1, c2, c3, c4} (3)

[0052] Therefore, any combination of the four EVS color channels used to obtain a given CIS color channel can be represented as follows:

[0053]

[0054] That is to say, the event integral of any CIS color channel p i is a function of the event integral of the event color channel c i .

[0055] The function of the event time integral can be shown as follows:

[0056]

[0057] , where f is the reference time and t represents the end time of event accumulation.

[0058] L(f) is the logarithm of the pixel illuminance of a given CIS color channel as a reference. The logarithm of the time-aligned image L(t) can be calculated as follows:

[0059] L(t) = L(f)exp(cE(t))

[0060] , where c is the event detection threshold.

[0061] Therefore, the CIS data at the reference time f and the event integral between time f and time t are combined to obtain the time estimate of a given color event pixel at time t.

[0062] Figure 3 Contains two time graphs 31 and 32. As Figure 3 shown, the vertical axis 311 of the time graph 31 is the logarithm of the pixel illuminance, and the vertical axis 321 of the time graph 32 is the event stream e(t). Each of the pulses in the time graph 32 means a detected event of one of the color event pixels. The detected events include positive events and negative events corresponding to intensity changes. For example, the pulse 323 is a positive event (e.g., an increase in light intensity) and the pulse 325 is a negative event (e.g., a decrease in light intensity).

[0063] The curve 313 in the timing diagram 31 is the actual pixel illuminance received by a given CIS color channel. The curve 315 in the timing diagram 31 is the reconstructed pixel illuminance of the given CIS color channel. As shown in the timing diagram 31, the curve 315 corresponds to the integration of the pulses in the timing diagram 32. At the time of each positive pulse, the curve 315 gradually increases the positive detection threshold c, and at the time of each negative pulse, the curve 315 gradually decreases the negative detection threshold c. As shown in the timing diagram 31, the curve 315 can fit the curve 313 such that the reconstructed pixel illuminance can be regarded as corresponding to the actual pixel illuminance received by the given CIS color channel.

[0064] In some embodiments of the present disclosure, this process can be applied to correct one or more color channels in an image sensor. In some embodiments of the present disclosure, this process can be applied to correct all color channels in an image sensor. In some embodiments of the present disclosure, this process can be applied to correct all pixel positions in an image sensor. In some embodiments of the present disclosure, this process can be applied to a specific region (e.g., region of interest) in an image sensor, which can be achieved by activity monitoring, object identification, or feature identification, etc.

[0065] In some embodiments of the present disclosure, input data formats other than event integration can be used for color space conversion. For example, events can be represented as voxels or pseudo-frames. The number of color channels of CIS and EVS can also be arbitrary and is not limited to three or four. That is, the CFA patterns of CIS and EVS can be arbitrary. In some embodiments of the present disclosure, in the case where the CFA pattern of EVS is the RGGB CFA pattern, the CFA pattern of CIS can be (but is not limited to) the RGGBCFA pattern or the RGBC CFA pattern. In some embodiments of the present disclosure, in the case where the CFA pattern of CIS is the RGGB CFA pattern, the CFA pattern of EVS can be (but is not limited to) the BCGR CFA pattern, the BYYR CFA pattern, or the CyYYMg CFA pattern. Other variations of the input data format can also include resolution (different or the same between CIS and EVS) and spatial arrangement. In some embodiments of the present disclosure, the spatial positions of the pixels to be used can depend on the color space conversion scheme.

[0066] Figure 4 The timing diagrams illustrate the logarithmic pixel illuminance and event stream according to some embodiments of the present disclosure.

[0067] Figure 3 Illustrate the reconstruction of the actual pixel illuminance using the detected events of one of the color event pixels. Figure 4 Further illustrate the reconstruction of the actual pixel illuminance using the detected events of more than one of the color event pixels. Figure 4includes four time graphs 41, 42, 43, and 44. Similar to Figure 3 , the vertical axis of time graph 41 is the logarithm of pixel illuminance, and the vertical axes of time graphs 42, 43, and 44 are event polarities as a function of time e(t). The pulses in time graphs 42, 43, and 44 represent Figure 1 the detected events of each of the color event pixels 111, 113, and 114 in

[0068] In one example, for a given color imaging pixel 101, a combination of nearby color EVS pixels (e.g., red EVS pixel data of color event pixel 111, green EVS pixel data of color event pixel 113, and blue EVS pixel data of color event pixel 114) can be used to obtain equivalent white (W) imaging data associated with the given color imaging pixel 101. The detected events accumulated between the reference time f and time t also include positive events and negative events. In one exemplary embodiment of the present disclosure, any combination of the three color event pixels can be shown as follows:

[0069] E W (t) = 0.25E R (t) + 0.5E G (t) + 0.25E B (t)

[0070] , where the weights 0.25, 0.5, and 0.25 are color weighting factors, which can be obtained based on an offline color calibration process, which will be explained in more detail later in Figure 6 .

[0071] As shown in time graph 42, during the time interval from the reference time f to time t, there are four positive events and one negative event. Therefore, assuming that the positive contrast detection threshold is the same as the negative contrast detection threshold, the total number of pulses counted within the time interval should be 3. The weight of the red event pixel is 0.25. Therefore, the weighted pulse of the time interval in time graph 42 is 0.75 (i.e., E R (t) = 0.25 * 3 = 0.75).

[0072] As shown in time graph 43, during the time interval from the reference time f to time t, there are five positive events and three negative events. Therefore, assuming that the positive contrast detection threshold is the same as the negative contrast detection threshold, the total number of pulses counted within the time interval should be 2. The weight of the green event pixel is 0.5. Therefore, the weighted pulse of the time interval in time graph 43 is 1 (i.e., E G(t) = 0.5 * 2 = 1).

[0073] As shown in timing diagram 44, during the time interval from time f to time t, there are two positive events and one negative event. Thus, assuming that the positive contrast detection threshold is the same as the negative contrast detection threshold, the total number of pulses counted within the time interval should be 1. The weight of the blue event pixels is 0.25. Thus, the weighted pulse for the time interval in timing diagram 44 is 0.25 (i.e., E B (t) = 0.25 * 1 = 0.25).

[0074] Thus, any combination of the three color event pixels E W (t) is 2 (i.e., E W (t) = 0.75 + 1 + 0.25). That is, as shown in timing diagram 41, curve 415 refers to the initial starting value at reference time f and gradually increases by two positive detection threshold c increments at time t.

[0075] Figure 5 is a flowchart 50 illustrating a method for converting color event data into color space data according to some embodiments of the present disclosure. Figure 5 Shows Figure 2 the details of step 206 in

[0076] The method starts from step 500. In some embodiments of the present disclosure, first, the destination of the CIS color channel (color imaging pixel) is defined in step 501. The target color to be calculated is also defined. In step 502, EVS pixels (color event pixels) that are closely close to the CIS color channel (associated with a given color imaging pixel) are selected and event data of the close selected EVS pixels are received from the EVS buffer as shown in Figure 2 . In step 503, a set of color conversion model parameters is retrieved from the register bank such that the set of color conversion model parameters can be applied to the event data of the EVS pixels to generate color space data. The register bank stores data from offline calibration, which will be explained in Figure 6 . Then, the color EVS data is converted to the destination CIS color space. In step 504, the color space data is returned for fusing the color space data with the image signal of the CIS color channel. The method ends at step 505. In some embodiments of the present disclosure, Figure 5 the method of Figure 2 can be executed in real time on the image sensor like the method of

[0077] Figure 6 is a flowchart 60 illustrating an error minimization process according to some embodiments of the present disclosure.

[0078] The error minimization process begins at step 600. In some embodiments of the present disclosure, at step 601, color CIS data is captured from color imaging pixels. At step 603, the color CIS data is stored in a CIS buffer. Similarly, in some embodiments of the present disclosure, at step 602, color EVS data is captured from color event pixels. At step 604, the color EVS data is stored in an EVS buffer. At step 605, the difference between the color CIS data of consecutive CIS frames is calculated. At step 607, patches of pixels where color changes in the color imaging pixels of different CIS frames are found. At step 606, the integral of the color EVS data over the time interval between corresponding CIS frames is calculated to obtain the accumulated color EVS information. Then, at step 610, the set of color conversion model parameters is optimized to minimize the error between the pixel value difference and the event integral, and at step 620, the set of color conversion model parameters is stored in a register bank. The error minimization process ends at step 630. In some embodiments of the present disclosure, Figure 6 the error minimization process is performed offline. That is, the color conversion model parameters are predetermined by the error minimization process such that the color conversion model parameters can be stored in a register bank for later use Figure 5 in the method of

[0079] Compared with Figure 2 and 5 the methods in Figure 2 and 5 the error minimization process is an offline calibration and Figure 2 and 5 the methods in

[0080] In some embodiments of the present disclosure, imaging different illuminants requires different sets of color conversion model parameters. In some embodiments of the present disclosure, the set of color conversion model parameters depends on the illumination level, the noise level, or the motion speed.

[0081] Figure 7 FIG. 70 is a schematic diagram illustrating an error minimization process according to some embodiments of the present disclosure.

[0082] Figure 7 is further explained in Figure 6 the error minimization process in Figure 7 Shows two consecutive CIS frames. CIS frame 701 contains pixels which represent pixels at coordinates (i, j) of CIS frame 701 at time t n . CIS frame 702 contains pixels which represent pixels at coordinates (i, j) of CIS frame 702 at time t n+1 . Time t n and time t n+1 are also shown on the horizontal axis 711 in Figure 7 to define a time period. As shown in box 703 and box 704, between time t n and time t n+1 , the color (e.g., red) of pixel changes to a different color (e.g., green) of pixel . On the horizontal axis 721 in Figure 7 is shown the pulse of the color EVS data during the time interval between time t n and time t n+1 . Color CIS data 731 is the color CIS data captured in CIS frame 701, and color CIS data 735 is the color CIS data captured in CIS frame 702. The difference between color CIS data 731 and color CIS data 735 is calculated. At the same time, the integral 732 of the color EVS data during the time interval between time t n and time t n+1 is also calculated. Applying the correct or appropriately configured integral 732 to color CIS data 731 can create reconstructed color CIS data 733. The difference between reconstructed color CIS data 733 and color CIS data 735 captured in CIS frame 702 at time t n+1 is error 734. In some embodiments of the present disclosure, the set of color conversion model parameters (e.g., color weighting factors) is optimized to minimize error 734.

[0083] In some embodiments of the present disclosure, the set of color conversion model parameters is in the form of one of the following conversion methods: linear combination, quadratic combination, neural network, or look-up table. In some embodiments of the present disclosure, the set of color conversion model parameters can be applied to the entire frame. In some embodiments of the present disclosure, the set of color conversion model parameters can only be applied to a local area (content-dependent).

[0084] Figure 8Schematic diagram of a system block diagram of an image system according to some embodiments of the present disclosure.

[0085] In some embodiments of the present disclosure, Figure 2 、 5 and the method in 6 can be executed by an image system 80 as shown in Figure 8 . In some embodiments of the present disclosure, the image system 80 includes a hybrid image sensor 81 and a host device 82. In some embodiments of the present disclosure, Figure 2 、 5 and the method in 6 can be executed by a chip - on - processor incorporated in the hybrid image sensor 81 of the imaging system 80. In some embodiments of the present disclosure, the hybrid image sensor 81 includes a CIS / EVS sensor core 811, a sensor processor 812, and an output interface 813. In some embodiments of the present disclosure, the CIS / EVS sensor core 811 includes an image array 8111, a row controller 8113, and a column controller 8115. In an embodiment, the image array 8111 is an example of the hybrid image sensor 10 of Figure 1 and the image array 8111 includes a plurality of color imaging pixels and a plurality of color event pixels for generating color signals and color event data in response to a captured scene. In some embodiments of the present disclosure, the row controller 8113 and the column controller 8115 respectively control the rows and columns of the pixels in the image array 8111. In an embodiment, the row controller 8113 and the column controller 8115 can be configured to generate control and timing signals for driving the operation of the image array 8111. In some embodiments of the present disclosure, the EVS data and CIS data output from the image array 8111 are transmitted to the sensor processor 812 to execute Figure 2 、 5 and the method in 6. In some embodiments of the present disclosure, the processed result can be transmitted to the output interface 813 for further transmission to the host device 82. In an embodiment, the external interface 813 can include a camera serial interface, a Mobile Industry Processor Interface (MIPI), and an Inter - Integrated Circuit (I2C) interface for communicating with the host device 82.

[0086] In some embodiments of the present disclosure, a sensor processor 812 and / or a portion of the CIS / EVS sensor core 811 may be configured to function as an EVS-to-CIS color space conversion unit. In some embodiments of the present disclosure, for latency and conversion efficiency considerations, the EVS-to-CIS color space conversion unit and a register bank 814 for storing one or more sets of color conversion model parameters (such as color weighting factors) for color optimization parameters may be included or otherwise located in the hybrid image sensor 81. In this embodiment, the register bank 814 includes one or more registers and the register bank 814 is coupled to the sensor processor 812. In some embodiments, the register bank may be allocated within a memory block of the sensor processor 812. In some embodiments of the present disclosure, the sensor processor 812 including the EVS-to-CIS color space conversion unit and the register bank is located in a host device 82 coupled to the hybrid image sensor 81 to relieve the storage and computing requirements of the hybrid image sensor 81. In still other embodiments of the present disclosure, the EVS-to-CIS color space conversion unit may be implemented by a processor programmed with firmware to perform a color imaging method, such as as illustrated by Figure 5 is illustrated.

[0087] The foregoing outlines the features of several embodiments so that those skilled in the art may better understand aspects of the present disclosure. Those skilled in the art should appreciate that they may readily use the present disclosure as a basis for designing or modifying other processes and structures for carrying out the same purposes and / or achieving the same advantages as the embodiments introduced herein. Those skilled in the art should also recognize that such equivalent constructs do not depart from the spirit and scope of the present disclosure, and that various changes, substitutions, and alterations may be made therein without departing from the spirit and scope of the present disclosure.

Claims

1. An image sensor, comprising: Multiple color imaging pixels; and Multiple color event pixels, An image signal of a first color imaging pixel included in the plurality of color imaging pixels is determined based on a first color signal of the first color image pixel and at least color event data of one or more color event pixels included in the plurality of color event pixels, wherein the color event data is generated in a manner that has a time relationship with the generation of the first color signal. 2 . The image sensor of claim 1 , wherein the one or more color event pixels are color event pixels of the plurality of color event pixels that are associated with the first color image pixel. 3 . The image sensor of claim 2 , wherein the one or more color event pixels are color event pixels of the plurality of color event pixels that are located in close proximity to the first color imaging pixel.

4. The image sensor of claim 1, wherein the color event data is generated during an integration time of the first color image pixel.

5. The image sensor of claim 4, wherein the color event data is generated within a time interval that is the same as the integration time of the first color image pixel. 6 . The image sensor according to claim 1 , wherein the color event data is generated in a time interval before and / or after a time when the first color signal is received.

7. The image sensor of claim 1, wherein a first color sensed by the first color image pixel is the same as a color of one of the one or more color event pixels.

8. The image sensor of claim 1, wherein a first color sensed by the first color image pixel is different from a color of each of the one or more color event pixels.

9. An image sensor according to claim 8, wherein the image signal of each individual color imaging pixel contained in the multiple color imaging pixels is determined based on the color signal of each corresponding color image pixel and the color event data of the one or more color event pixels contained in the multiple color event pixels.

10. The image sensor of claim 1, wherein the one or more color event pixels are associated with each color imaging pixel of the plurality of color imaging pixels.

11. The image sensor of claim 1, wherein the first color imaging pixel included in a plurality of color image pixels is located within a region of interest of a pixel array of the image sensor.

12. A color imaging method, the method comprising: defining a target complementary metal oxide semiconductor image sensor color channel; selecting one or more event-based vision sensor pixels depending on proximity to the target complementary metal oxide semiconductor image sensor color channel; applying a set of color conversion model parameters to event data associated with the one or more event-based vision sensor pixels to generate color space data, wherein the event data associated with the one or more event-based vision sensor pixels is captured in a time interval before or after a time when a color image signal is received; and The color space data is fused with the color image signal of the target complementary metal oxide semiconductor image sensor color channel.

13. The method of claim 12, wherein the set of color conversion model parameters are stored in a register bank.

14. The method of claim 13, wherein the set of color conversion model parameters including a set of weight factors associated with the one or more event-based vision sensor pixels are predetermined by an error minimization process.

15. The method of claim 14, wherein the error minimization process comprises: calculating differences between color CMOS image sensor data of consecutive CMOS image sensor frames; calculating an integral of color event-based vision sensor data over a time interval between consecutive CMOS image sensor frames; The set of color conversion model parameters is optimized to minimize an error between the difference and the integral.

16. The method of claim 12, wherein imaging different illuminants requires different sets of color conversion model parameters.

17. The method of claim 12, wherein the set of color conversion model parameters depends on illumination level, noise level, or motion speed.

18. The method of claim 15, wherein the set of color conversion model parameters is in the form of one of a linear combination, a quadratic combination, a neural network, or a lookup table.

19. An event-based vision sensor to complementary metal oxide semiconductor image sensor color space conversion unit configured to operatively perform a color imaging method, the color imaging method comprising: defining a target complementary metal oxide semiconductor image sensor color channel; selecting one or more event-based vision sensor pixels depending on proximity to the target complementary metal oxide semiconductor image sensor color channel; applying a set of color conversion model parameters to the event-based vision sensor data of the one or more event-based vision sensor pixels to generate color space data; and The generated color space data is output.

20. The event-based vision sensor to complementary metal oxide semiconductor image sensor color space conversion unit of claim 19, wherein the set of color conversion model parameters are stored in a register bank.

21. The event-based vision sensor to complementary metal oxide semiconductor image sensor color space conversion unit according to claim 20, wherein the event-based vision sensor to complementary metal oxide semiconductor image sensor color space conversion unit and the register library are included in an image sensor, and the image sensor has a pixel array including a plurality of color imaging pixels and a plurality of color event pixels.

22. The event-based vision sensor to complementary metal oxide semiconductor image sensor color space conversion unit of claim 20, wherein the event-based vision sensor to complementary metal oxide semiconductor image sensor color space conversion unit and the register bank are included in a host device connected to an image sensor, the image sensor having a pixel array including a plurality of color imaging pixels and a plurality of color event pixels.

Images