Imager health monitoring system and method

Abstract

Technologies for imager health monitoring systems and methods are disclosed. In one example, a method includes determining a characteristic of an active unit cell of a focal plane array (FPA) and / or a reference unit cell of the FPA. The active unit cell includes a detector that is selectively shielded from an incident scene. The reference unit cell includes a reference detector that is shielded from an incident scene. The method also includes determining a state of the FPA based at least in part on the characteristic. The method also includes transmitting an indication of the state of the FPA to selectively cause an adjustment of the FPA. Related devices and systems are also provided.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to and the benefit of U.S. Provisional Patent Application No. 62 / 849,479, filed on May 17, 2019, and entitled “IMAGERHEALTH MONITORING SYSTEMS AND METHODS,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] One or more embodiments relate generally to imaging systems and, more particularly, for example, to imager health monitoring systems and methods. Background Art

[0004] An imaging system may include an array of detectors, each detector being used as a portion of a pixel to produce a two-dimensional image. There are a variety of image detectors, such as visible light image detectors, infrared image detectors, or other types of image detectors that may be arranged in an image detector array for capturing an image. For example, a plurality of sensors may be arranged in an image detector array to detect electromagnetic (EM) radiation of a desired wavelength. In some cases, such as for infrared imaging, readout of image data captured by the detector may be performed in a time-division multiplexed manner by a readout integrated circuit (ROIC). The readout image data may be transmitted to other circuits, such as for processing, storage, and / or display. In some cases, the combination of a detector array and a ROIC may be referred to as a focal plane array (FPA). Advances in FPA and image processing process technology have increased the capabilities and complexity of the resulting imaging systems. Summary of the invention

[0005] In one or more embodiments, a method includes determining a characteristic of at least one active unit cell of a focal plane array and / or at least one reference unit cell of the focal plane array. At least one active unit cell includes a detector that is selectively shielded from an incident scene. At least one reference unit cell includes a reference detector that is shielded from an incident scene. The method also includes determining a state of the focal plane array based at least in part on the characteristic. The method also includes transmitting an indication of the state of the focal plane array to selectively cause an adjustment of the focal plane array.

[0006] In one or more embodiments, an imaging system includes a focal plane array. The focal plane array includes a plurality of effective unit cells. Each effective unit cell includes a detector configured to selectively generate an output in response to an incident scene. The focal plane array also includes a plurality of reference unit cells. Each reference unit cell includes a detector and a structure configured to block the incident scene from being received by the detector of the reference unit cell. The imaging system also includes a processing circuit configured to determine a characteristic of at least one of the plurality of effective unit cells and / or at least one of the plurality of reference unit cells. The processing circuit is also configured to determine a state of the focal plane array based at least in part on the characteristic. The processing circuit is also configured to transmit an indication of the state of the focal plane array to selectively cause adjustment of the focal plane array.

[0007] In one or more embodiments, a method includes receiving an image including data and a first hash value. The method also includes generating a second hash value based on the data. The method also includes determining the validity of the image based on the first hash value and the second hash value.

[0008] The scope of the present disclosure is defined by the claims, which are incorporated into this section by reference. A more complete understanding of the embodiments of the present disclosure will be provided to those skilled in the art, as well as realizing additional advantages thereof, by considering the following detailed description of one or more embodiments. Reference will be made to the accompanying drawings, which will first be briefly described. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A block diagram of an example imaging system is shown in accordance with one or more embodiments of the present disclosure.

[0010] Figure 2A A block diagram of an example image sensor assembly is shown in accordance with one or more embodiments of the present disclosure.

[0011] Figure 2B An example image sensor assembly is shown in accordance with one or more embodiments of the present disclosure.

[0012] Figure 3A , 3B 3C and 3C respectively represent the power spectral densities of the fixed pattern noise, the random noise, and the weighted sum of the fixed pattern noise and the random noise according to one or more embodiments of the present disclosure.

[0013] Figure 4 A circuit diagram showing an example of a circuit for measuring the resistance of a microbolometer according to one or more embodiments of the present disclosure and a side view of a microbolometer are shown.

[0014] Figure 5An example graph showing the change in resistance of a bolometer as a function of time is shown in accordance with one or more embodiments of the present disclosure.

[0015] Figure 6 An example array of active unit cells and potential locations of one or more reference unit cells are shown in accordance with one or more embodiments of the present disclosure.

[0016] Figure 7 Example image frame formats according to one or more embodiments of the present disclosure are shown.

[0017] Figure 8 A flow chart illustrating an example of a process for facilitating monitoring the health of an imaging device in accordance with one or more embodiments of the present disclosure.

[0018] Fig. 9 A flow chart illustrating an example of a process for generating an image according to one or more embodiments of the present disclosure is shown.

[0019] Fig.10 A flow chart illustrating an example of a process for determining the validity of a received image in accordance with one or more embodiments of the present disclosure.

[0020] Embodiments of the present disclosure and their advantages may be best understood by reference to the detailed description below. It should be noted that the sizes of the various components and the distances between these components are not drawn to scale in the drawings. It should be understood that similar reference numerals are used to identify similar elements shown in one or more figures. DETAILED DESCRIPTION

[0021] The detailed description set forth below is intended as a description of various configurations of the subject technology and is not intended to represent the only configuration in which the subject technology can be practiced. The accompanying drawings are incorporated herein and constitute a part of the detailed description. The detailed description includes specific details for providing a thorough understanding of the subject technology. However, it is clear and obvious to those skilled in the art that the subject technology is not limited to the specific details set forth herein and can be practiced using one or more embodiments. In one or more instances, the structures and components are shown in block diagram form to avoid blurring the concept of the subject technology. One or more embodiments disclosed in the subject are illustrated by one or more diagrams and / or described in conjunction with one or more diagrams and set forth in the claims. It should be noted that the sizes of the various components in the drawings are not necessarily drawn to scale.

[0022] In one or more embodiments, various systems and methods are provided to facilitate monitoring the health of an imaging device. An imaging device may include a focal plane array (FPA) including an active unit cell and a reference unit cell. Monitoring the health of the imaging device may be based on characteristics of the active unit cells of the imaging device and / or corresponding characteristics of the reference unit cells of the imaging device. As an example, monitoring the health of the imaging device may be based on a comparison of characteristics of the active unit cells with corresponding characteristics of the reference unit cells, such as a comparison of noise characteristics. In some aspects, image frames transmitted by the imaging device may utilize an image frame format that facilitates verification of data contained in the image frames.

[0023] Each effective unit cell may include a detector (e.g., also referred to as a pixel). The detector of each effective unit cell may capture a portion of the incident scene and generate an associated output. Each effective unit cell has a structure (e.g., a shutter, a cover, a cover, a blade) or a portion thereof that selectively blocks the detector of the effective unit cell from receiving the scene incident on the effective unit cell. In some cases, the structure provides / presents a uniform scene to the detector. Each reference unit cell may include a detector and a structure (e.g., a shutter, a cover, a cover, a blade) or a portion thereof that blocks incident or external radiation (e.g., thermal radiation) from being received by the detector. At this point, the structure blocks the detector from receiving the scene incident on the effective unit cell. In some cases, the reference unit cell may be outside the effective array including the effective unit cell and behaves similarly to the effective detector, except that the reference unit cell is shielded from the scene and is therefore actually invisible to the scene. Therefore, the reference unit cell and its detector may be referred to as a blind unit cell and a blind detector, respectively. In an embodiment, each effective unit cell and the reference unit cell include a bolometer (e.g., a micro-side bolometer) as their detector.

[0024] In some aspects, the imaging device can be part of a larger system. In some cases, the larger system can include one or more additional sensors. For purposes of explanation, an imaging device can be used in an advanced driver assistance system (ADAS) for an automotive vehicle (AV). In the automotive industry, imaging systems can be used in ADAS and AV for safety-critical applications such as braking, steering, and acceleration. There are different ratings for Automotive Safety Integrity Levels (ASILs) depending on the criticality of the function. For example, an ASIL rating can be determined by the severity of a fault, the probability of a fault occurring, and the ability to detect and control the impact. It should be noted that while various embodiments are described with respect to the automotive industry, the imaging device may be used alone (e.g., not part of any larger system) or as part of one or more systems not related to the automotive industry.

[0025] In various embodiments, fault detection may be included in the imaging device (e.g., rather than implementing a fault detection algorithm downstream of the imaging device) to reduce a fault detection time interval (FTDI). Detection of potential faults in the imaging device and communication thereof to the safety system may facilitate mitigation of such faults and may allow a host (e.g., an ADAS host system or its user) to make decisions to evaluate and ensure the safety of a system (e.g., an AV system).

[0026] Now referring to the accompanying drawings, Figure 1 A block diagram of an example imaging system 100 (e.g., an infrared camera) according to one or more embodiments of the present disclosure is shown. However, not all of the components depicted are required, and one or more embodiments may include additional components not shown in the figure. The arrangement and type of components may be changed without departing from the spirit or scope of the claims set forth herein. Additional components, different components, and / or fewer components may be provided.

[0027] According to embodiments of the present disclosure, imaging system 100 may be used to capture and process images. Imaging system 100 may represent any type of imaging system that detects one or more ranges (e.g., frequency bands) of electromagnetic (EM) radiation and provides representative data (e.g., one or more still image frames or video image frames). Imaging system 100 may include a housing that at least partially encloses components of imaging system 100, such as to facilitate compactness and protection of imaging system 100. For example, Figure 1 The solid line frame marked as 100 in the figure may represent the housing of the imaging system 100. The housing may include Figure 1 Components depicted within the solid line box are more, fewer, and / or different components of the imaging system 100. In embodiments, the imaging system 100 may include a portable device and may be incorporated into, for example, a vehicle or non-mobile device where storage and / or display of images is desired. The vehicle may be a land-based vehicle (e.g., a car), a sea-based vehicle, an aircraft (e.g., an unmanned aerial vehicle (UAV)), a space vehicle, or generally any other type of vehicle that may incorporate (e.g., be mounted therein, mounted thereon, etc.) the imaging system 100. In another example, the imaging system 100 may be coupled to various types of fixed locations (e.g., a home security mount, a campsite or outdoor mount, or other locations) via one or more types of mounts.

[0028] According to one embodiment, the imaging system 100 includes a processing component 105, a memory component 110, an image capture component 115, an image interface 120, a control component 125, a display component 130, a sensing component 135, and / or a network interface 140. According to various embodiments, the processing component 105 includes one or more of a processor, a microprocessor, a central processing unit (CPU), a graphics processing unit (GPU), a single-core processor, a multi-core processor, a microcontroller, a programmable logic device (PLD) (e.g., a field programmable gate array (FPGA)), an application specific integrated circuit (ASIC), a digital signal processing (DSP) device, or other logic devices that can be configured to perform various operations discussed herein for embodiments of the present disclosure by hardwiring, executing software instructions, or a combination of both. The processing component 105 can be configured to connect and communicate with various other components of the imaging system 100 (e.g., 110, 115, 120, 125, 130, 135, etc.) to perform such operations. For example, processing component 105 can be configured to process captured image data received from image capture component 115, store the image data in memory component 110, and / or retrieve stored image data from memory component 110. In one aspect, processing component 105 can be configured to perform various system control operations (e.g., controlling the communications and operations of various components of imaging system 100) and other image processing operations (e.g., data conversion, video analysis, etc.).

[0029] In one embodiment, the memory component 110 includes one or more memory devices configured to store data and information, including infrared image data and information. The memory component 110 may include one or more different types of memory devices, including volatile and non-volatile memory devices, such as random access memory (RAM), dynamic RAM (DRAM), static RAM (SRAM), non-volatile random access memory (NVRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, hard disk drive, and / or other types of memory. As described above, the processing component 105 can be configured to execute software instructions stored in the memory component 110 to perform method and process steps and / or operations. The processing component 105 and / or the image interface 120 can be configured to store images or digital image data captured by the image capture component 115 in the memory component 110. The processing component 105 can be configured to store processed static and / or video images in the memory component 110. In some aspects, memory component 110 may include information that may be used as a reference point for comparison purposes for determining the health of imaging system 100. As non-limiting examples, this information may include characteristics associated with image detector circuit 165, such as power spectral density, thermal time constants, etc.

[0030] In some embodiments, a separate machine-readable medium 145 (e.g., a memory such as a hard drive, compact disk, digital video disk, or flash memory) can store software instructions and / or configuration data that can be executed or accessed by a computer (e.g., a logic device or processor-based system) to perform various methods and operations, such as those associated with processing image data. In one aspect, the machine-readable medium 145 can be portable and / or located separately from the imaging system 100, with the stored software instructions and / or data provided to the imaging system 100 by coupling the machine-readable medium 145 to the imaging system 100 and / or downloading from the machine-readable medium 145 by the imaging system 100 (e.g., via a wired link and / or a wireless link). It should be understood that the various modules can be integrated in software and / or hardware as part of the processing component 105, wherein the code (e.g., software or configuration data) of the modules is stored, for example, in the memory component 110.

[0031] The imaging system 100 may represent an imaging device, such as a video and / or still camera, to capture and process images and / or video of a scene 160. In this regard, the image capture component 115 of the imaging system 100 may be configured to capture images (e.g., still and / or video images) of the scene 160 in a particular spectrum or modality. The image capture component 115 includes an image detector circuit 165 (e.g., a thermal infrared detector circuit) and a readout circuit 170 (e.g., a ROIC). In some aspects, the image detector circuit 165 includes an active unit cell and a reference unit cell. For example, the image capture component 115 may include an IR imaging sensor (e.g., an IR imaging sensor array) configured to detect IR radiation in the near, mid, and / or far IR spectrum and provide an IR image (e.g., IR image data or signal) representing IR radiation from the scene 160. For example, the image detector circuit 165 may capture (e.g., detect, sense) IR radiation having a wavelength in a range from about 700 nm to about 2 mm or a portion thereof. For example, in some aspects, image detector circuit 165 can be sensitive to (e.g., better detect) short wave IR (SWIR) radiation, medium wave IR (MWIR) radiation (e.g., EM radiation with a wavelength of 2-5 μm), and / or long wave IR (LWIR) radiation (e.g., EM radiation with a wavelength of 7-14 μm), or any desired IR wavelength (e.g., generally in the range of 0.7 to 14 μm). In other aspects, image detector circuit 165 can capture radiation from one or more other bands of the EM spectrum, such as visible light, ultraviolet light, etc.

[0032] Image detector circuit 165 can capture image data associated with scene 160. To capture an image, image detector circuit 165 can detect image data of scene 160 (e.g., in the form of EM radiation) and generate pixel values ​​of the image based on scene 160. An image can be referred to as a frame or image frame. In some cases, image detector circuit 165 can include a detector array (e.g., also referred to as a pixel array) that can detect radiation of a specific frequency band, convert the detected radiation into an electrical signal (e.g., voltage, current, etc.), and generate pixel values ​​based on the electrical signal. Each detector in the array can capture a corresponding portion of the image data and generate a pixel value based on the corresponding portion captured by the detector. In one aspect, each detector has an active area for capturing a corresponding portion of the image data. The inactive area of ​​the detector can refer to any portion of the detector that is not an active area. The pixel value generated by the detector can be referred to as an output of the detector. As a non-limiting example, each detector can be a photodetector, such as an avalanche photodiode, an infrared photodetector, a quantum well infrared photodetector, a microbolometer, or other detectors capable of converting EM radiation (e.g., EM radiation of a specific wavelength) into pixel values. The detector array can be arranged in rows and columns. Although the present disclosure generally refers to various operations performed on rows and / or columns, rows may be used as columns and columns may be used as rows where appropriate.

[0033] In one aspect, imaging system 100 (e.g., image capture component 115 of imaging system 100) can include one or more optical elements (e.g., mirrors, lenses, beam splitters, beam couplers, etc.) to direct EM radiation to image detector circuit 165. In some cases, the optical elements can be at least partially located within a housing of imaging system 100.

[0034] An image may be or may be considered to be a data structure including pixels and is a representation of image data associated with scene 160, wherein each pixel has a pixel value representing EM radiation emitted or reflected from a portion of the scene and received by a detector that generates the pixel value. Depending on the context, a pixel may refer to a detector of image detector circuitry 165 that generates an associated pixel value or a pixel (e.g., pixel position, pixel coordinates) of an image formed from the generated pixel values.

[0035] In an embodiment, the pixel value generated by the image detector circuit 165 can be represented according to a digital count value generated based on an electrical signal obtained from converting the detected radiation. For example, where the image detector circuit 165 includes or is otherwise coupled to an ADC circuit, the ADC circuit can generate a digital count value based on the electrical signal. For an ADC circuit that can use 14 bits to represent the electrical signal, the digital count value can range from 0 to 16383. In such a case, the pixel value of the detector can be a digital count value output from the ADC circuit. In other cases (e.g., in the absence of an ADC circuit), the pixel value can be analog in nature, and its value is or indicates the value of the electrical signal. For example, for infrared imaging, a greater amount of IR radiation incident on and detected by the image detector circuit 165 (e.g., an IR image detector circuit) is associated with a higher digital count value and a higher temperature.

[0036] The image detector circuit 165 may be coupled to the readout circuit 170. The readout circuit 170 may be disposed on a substrate formed of, for example, silicon, silicon carbide, gallium arsenide, germanium, or any other material suitable for supporting the integration of semiconductor devices. The readout circuit 170 may serve as an interface between the image detector circuit 165 that detects image data and the processing component 105 that processes the detected image data read out by the readout circuit 170, wherein the transfer of data from the readout circuit 170 to the processing component 105 is facilitated by the image interface 120. The image capture frame rate may refer to the rate (e.g., the number of images per second) at which the image detector circuit 165 sequentially detects images and provides the images to the processing component 105 via the readout circuit 170. The readout circuit 170 may read out the pixel values ​​generated by the image detector circuit 165 according to an integration time (e.g., also referred to as an integration period). In some cases, an ADC circuit for generating a digital count value for a detector of the image detector circuit 165 may be implemented as part of the image detector circuit 165. In other cases, such an ADC circuit may be implemented as part of the readout circuit 170.

[0037] In various embodiments, the combination of image detector circuit 165 and readout circuit 170 may be, may include, or may together provide an FPA. In some aspects, image detector circuit 165 may be a thermal image detector circuit including an array of microbolometers, and the combination of image detector circuit 165 and readout circuit 170 may be referred to as a microbolometer FPA. In some cases, the array of microbolometers may be arranged in rows and columns. The microbolometers may detect IR radiation and generate pixel values ​​based on the detected IR radiation. For example, in some cases, the microbolometers may be thermal IR detectors that detect IR radiation in the form of thermal energy and generate pixel values ​​based on the detected thermal energy. The microbolometer FPA may include IR detecting materials, such as amorphous silicon (a-Si), vanadium oxide (VO2), or the like. x ), combinations thereof, and / or other detection materials. In one aspect, for a microbolometer FPA, the integration time may be or may indicate a time interval at which the microbolometer is biased. In this case, a longer integration time may be associated with a higher IR signal gain, but more IR radiation may not be collected. Infrared radiation may be collected by a microbolometer in the form of heat energy.

[0038] In some cases, the image capture component 115 may include one or more filters that are adapted to pass some wavelengths of radiation but substantially block other wavelengths of radiation. For example, the image capture component 115 may be an IR imaging device that includes one or more filters that are adapted to pass some wavelengths of IR radiation while substantially blocking other wavelengths of IR radiation (e.g., MWIR filters, thermal IR filters, and narrowband filters). In this example, such filters can be used to customize the image capture component 115 to increase sensitivity to a desired IR wavelength band. In one aspect, when the IR imaging device is customized for capturing thermal IR images, the IR imaging device may be referred to as a thermal imaging device. Other imaging devices, including IR imaging devices customized for capturing infrared IR images outside the thermal range, may be referred to as non-thermal imaging devices.

[0039] In one specific non-limiting example, the image capture component 115 may include an IR imaging sensor having a detector FPA responsive to radiation including near infrared (NIR), SWIR, MWIR, LWIR, and / or very long wave infrared (VLWIR). In some other embodiments, alternatively or additionally, the image capture component 115 may include a complementary metal oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor, which may be found in any consumer-grade camera (e.g., a visible light camera).

[0040] Other imaging sensors that may be included in the image capture component 115 include photonic mixer device (PMD) imaging sensors or other time-of-flight (ToF) imaging sensors, light detection and ranging (LIDAR) imaging devices, millimeter imaging devices, positron emission tomography (PET) scanners, single photon emission computed tomography (SPECT) scanners, ultrasound imaging devices, or other imaging devices operating in a specific modality and / or spectrum. It is noteworthy that for some of these imaging sensors that are configured to capture images in a specific modality and / or spectrum (e.g., infrared spectrum, etc.), for example, when compared to general CMOS-based or CCD-based imaging sensors or other imaging sensors, imaging scanners, or imaging devices of different modalities, they are more likely to produce images with low-frequency shadows.

[0041] The image provided by the image capture component 115 or the digital image data corresponding to the image can be associated with a corresponding image size (also referred to as pixel size). Image size or pixel size generally refers to the number of pixels in the image, which can be represented, for example, as the width multiplied by the height of a two-dimensional image or otherwise suitable for the relevant size or shape of the image. Therefore, an image with original resolution can be resized to a smaller size (e.g., with a smaller pixel size) to, for example, reduce the cost of processing and analyzing the image. A filter (e.g., non-uniformity estimation) can be generated based on an analysis of the resized image. The filter can then be resized to the original resolution and size of the image before being applied to the image.

[0042] In some embodiments, the image interface 120 may include appropriate input ports, connectors, switches, and / or circuits configured to connect to an external device (e.g., remote device 150 and / or other device) to receive an image (e.g., digital image data) generated by or otherwise stored at the external device. The received image or image data may be provided to the processing component 105. At this point, the received image or image data may be converted into a signal or data suitable for processing by the processing component 105. For example, in one embodiment, the image interface 120 may be configured to receive analog video data and convert it into suitable digital data to provide to the processing component 105.

[0043] In some embodiments, the image interface 120 may include various standard video ports that may be connected to a video player, a camera, or other device capable of generating a standard video signal, and may convert the received video signal into digital video / image data suitable for processing by the processing component 105. In some embodiments, the image interface 120 may also be configured to interface with the image capture component 115 and receive an image (e.g., image data) from the image capture component 115. In other embodiments, the image capture component 115 may interface directly with the processing component 105.

[0044] In one embodiment, the control component 125 includes a user input and / or interface device, such as a rotatable knob (e.g., a potentiometer), a button, a slide bar, a keyboard, and / or other device, which is suitable for generating a user input control signal. The processing component 105 can be configured to sense a control input signal from a user via the control component 125, and respond to any sensed control input signal received therefrom. As is generally understood by those skilled in the art, the processing component 105 can be configured to interpret such a control input signal as a value. In one embodiment, the control component 125 can include a control unit (e.g., a wired or wireless handheld control unit) having buttons suitable for interacting with a user and receiving user input control values. In one embodiment, the buttons of the control unit can be used to control various functions of the imaging system 100, such as autofocus, menu activation and selection, field of view, brightness, contrast, noise filtering, image enhancement, and / or various other functions of the imaging system or camera.

[0045] In one embodiment, the display component 130 includes an image display device (e.g., a liquid crystal display (LCD)) or various other types of well-known video displays or monitors. The processing component 105 can be configured to display image data and information on the display component 130. The processing component 105 can be configured to acquire image data and information from the memory component 110 and display any acquired image data and information on the display component 130. The display component 130 can include display circuitry that the processing component 105 can utilize to display image data and information. The display component 130 can be adapted to receive image data and information directly from the image capture component 115, the processing component 105, and / or the image interface 120, or the image data and information can be transmitted from the memory component 110 via the processing component 105.

[0046] In one embodiment, the sensing component 135 includes one or more sensors of various types, as will be appreciated by those skilled in the art, depending on the application or implementation requirements. The sensors of the sensing component 135 provide data and / or information to at least the processing component 105. In one aspect, the processing component 105 may be configured to communicate with the sensing component 135. In various embodiments, the sensing component 135 may provide information about environmental conditions, such as outdoor temperature, lighting conditions (e.g., daytime, nighttime, dusk, and / or dawn), humidity levels, specific weather conditions (e.g., sun, rain, and / or snow), distances (e.g., laser rangefinders or time-of-flight cameras), and / or whether a tunnel or other type of enclosure is entered or exited. The sensing component 135 may represent conventional sensors known to those skilled in the art for monitoring various conditions (e.g., environmental conditions) that may have an impact on the image data provided by the image capture component 115 (e.g., on the appearance of the image).

[0047] In some embodiments, sensing component 135 (e.g., one or more sensors) may include devices that relay information to processing component 105 via wired and / or wireless communications. For example, sensing component 135 may be adapted to receive information from satellites via local broadcast (e.g., radio frequency (RF)) transmissions, via mobile or cellular networks, and / or via information beacons in infrastructure (e.g., traffic or highway information beacon infrastructure), or various other wired and / or wireless technologies. In some embodiments, processing component 105 may use information obtained from sensing component 135 (e.g., sensory data) to modify the configuration of image capture component 115 (e.g., adjust light sensitivity levels, adjust the orientation or angle of image capture component 115, adjust aperture, etc.).

[0048] In some embodiments, the various components of the imaging system 100 may be distributed over the network 155 and communicate with each other. In this regard, the imaging system 100 may include a network interface 140 configured to facilitate wired and / or wireless communications between the various components of the imaging system 100 over the network 155. In such embodiments, components may also be replicated if required by a particular application of the imaging system 100. That is, components configured for the same or similar operations may be distributed over the network. In addition, if desired, all or part of any of the various components may be implemented using components of a remote device 150 (e.g., a conventional digital video recorder (DVR), a computer configured for image processing, and / or other device) that communicates with the various components of the imaging system 100 over the network 155 via the network interface 140. Thus, for example, all or part of the processing component 105, all or part of the memory component 110, and / or all or part of the display component 130 may be implemented or replicated at the remote device 150. In some embodiments, the imaging system 100 may not include an imaging sensor (e.g., image capture component 115), but rather receive images or image data from an imaging sensor that is located separate and remote from the processing component 105 and / or other components of the imaging system 100. It should be understood that many other combinations of distributed implementations of the imaging system 100 are possible without departing from the scope and spirit of the present disclosure.

[0049] Furthermore, in various embodiments, various components of the imaging system 100 may be combined and / or implemented or not implemented as desired or depending on the application or requirements. In one example, the processing component 105 may be combined with the memory component 110, the image capture component 115, the image interface 120, the display component 130, the sensing component 135, and / or the network interface 140. In another example, the processing component 105 may be combined with the image capture component 115 such that certain functions of the processing component 105 are performed by circuitry (e.g., a processor, microprocessor, logic device, microcontroller, etc.) within the image capture component 115.

[0050] Figure 2A A block diagram of an example image sensor assembly 200 is shown in accordance with one or more embodiments of the present disclosure. However, not all of the depicted components are required, and one or more embodiments may include additional components not shown in the figure. Changes may be made to the arrangement and types of components without departing from the spirit or scope of the claims set forth herein. Additional components, different components, and / or fewer components may be provided. In embodiments, the image sensor assembly 200 may be an FPA, for example, implemented as Figure 1 The image capturing component 115 is provided.

[0051] The image sensor assembly 200 includes a unit cell array 205, column multiplexers 210 and 215, column amplifiers 220 and 225, a row multiplexer 230, a control bias and timing circuit 235, a digital-to-analog converter (DAC) 240, and a data output buffer 245. The unit cell array 205 includes an array of unit cells. In one aspect, each unit cell can include a detector and an interface circuit. The interface circuit of each unit cell can provide an output signal, such as an output voltage or an output current, in response to a detector signal (e.g., a detector current, a detector voltage) provided by a detector of the unit cell. The output signal can indicate a magnitude of EM radiation received by the detector. The column multiplexer 215, the column amplifier 220, the row multiplexer 230, and the data output buffer 245 can be used to provide an output signal from the unit cell array 205 as a data output signal on a data output line 250. The output signal on the data output line 250 can be provided to components downstream of the image sensor assembly 200, such as a processing circuit (e.g., Figure 1 105), memory (e.g., Figure 1 110), a display device (e.g., Figure 1 The data output signal may be an image formed by the pixel values ​​of the image sensor assembly 200. In this regard, the column multiplexer 215, the column amplifier 220, the row multiplexer 230, and the data output buffer 245 may collectively provide the ROIC (or a portion thereof) of the image sensor assembly 200.

[0052] The column amplifier 225 may generally represent any column processing circuit suitable for a given application (analog and / or digital), and is not limited to amplifier circuits for analog signals. In this regard, in such an aspect, the column amplifier 225 may be more generally referred to as a column processor. The signal received by the column amplifier 225, such as an analog signal on an analog bus and / or a digital signal on a digital bus, may be processed according to the analog or digital nature of the signal. For example, the column amplifier 225 may include a circuit for processing a digital signal. As another example, the column amplifier 225 may be a path (e.g., without processing) that a digital signal from the unit cell array 205 passes through to reach the column multiplexer 215. As another example, the column amplifier 225 may include an ADC for converting an analog signal into a digital signal (e.g., to obtain a digital count value). These digital signals may be provided to the column multiplexer 215.

[0053] Each unit cell may receive a bias signal (e.g., a bias voltage, a bias current) to bias the detector of the unit cell to compensate for different response characteristics of the unit cell due to, for example, temperature variations, manufacturing variations, and / or other factors. For example, the control bias and timing circuit 235 may generate bias signals and provide them to the unit cell. By providing appropriate bias signals to each unit cell, the unit cell array 205 may be effectively calibrated to provide accurate image data in response to light (e.g., IR light) incident on the detector of the unit cell.

[0054] In one aspect, the control bias and timing circuit 235 may generate a bias value, a timing control voltage, and a switch control voltage. In some cases, the DAC 240 may convert a bias value received as a data input signal on the data input signal line 255 or as part of the data input signal into a bias signal (e.g., an analog signal on the analog signal line 260), which may be provided to individual unit cells through the operation of the column multiplexer 210, the column amplifier 220, and the row multiplexer 230. In another aspect, the control bias and timing circuit 235 may generate a bias signal (e.g., an analog signal) and provide the bias signal to the unit cell without using the DAC 240. In this regard, some embodiments do not include the DAC 240, the data input signal line 255, and / or the analog signal line 260. In an embodiment, the control bias and timing circuit 235 may be Figure 1 The processing component 105 and / or the image capturing component 115 may include Figure 1 The processing component 105 and / or the image capturing component 115 may be Figure 1 The processing component 105 and / or the image capture component 115 may be a portion of or otherwise coupled to Figure 1 processing component 105 and / or image capture component 115.

[0055] In one embodiment, the image sensor assembly 200 can be implemented as part of an imaging system (e.g., 100). In addition to the various components of the image sensor assembly 200, the imaging system can also include one or more processors, memories, logic, displays, interfaces, optical devices (e.g., lenses, mirrors, beam splitters), and / or other components that may be suitable for various embodiments. In one aspect, the data output signal on the data output line 250 can be provided to a processor (not shown) for further processing. For example, the data output signal can be an image formed by pixel values ​​of unit cells from the image sensor assembly 200. The processor can perform operations such as non-uniformity correction (NUC), spatial and / or temporal filtering, and / or other operations. The image (e.g., a processed image) can be stored in a memory (e.g., external or local to the imaging system) and / or displayed on a display device (e.g., external to the imaging system and / or integrated with the imaging system).

[0056] As non-limiting examples, the unit cell array 205 may include 512×512 (e.g., 512 rows and 512 columns of unit cells), 1024×1024, 2048×2048, 4096×4096, 8192×8192, and / or other array sizes. In some cases, the array size may have a row size (e.g., the number of detectors in a row) that is different from the column size (e.g., the number of detectors in a column). Examples of frame rates may include 30 Hz, 60 Hz, and 120 Hz. In one aspect, each unit cell of the unit cell array 205 may represent a pixel. In one aspect, the unit cell array 205 may include an effective unit cell and a reference unit cell. In another aspect, the unit cell array 205 may include only effective unit cells. In this regard, the reference unit cell (if present) may be considered to be separate from the unit cell array 205.

[0057] In one embodiment, the components of image sensor assembly 200 may be implemented such that the detector array is blended into (eg, coupled to) the readout circuitry. For example, Figure 2B An example image sensor assembly 265 is shown in accordance with one or more embodiments of the present disclosure. However, not all of the depicted components are required, and one or more embodiments may include additional components not shown in the figures. The arrangement and types of components may be changed without departing from the spirit or scope of the claims set forth herein. Additional components, different components, and / or fewer components may be provided. In embodiments, image sensor assembly 265 may be, may include, or may be a part of image sensor assembly 200.

[0058] The image sensor assembly 265 includes a device wafer 270, a readout circuit 275, and contacts 280 to bond (e.g., mechanically and electrically bond) the device wafer 270 to the readout circuit 275. The device wafer 270 may include a detector (e.g., a unit cell array 205). The contacts 280 may bond the detector of the device wafer 270 and the readout circuit 275. The contacts 280 may include conductive contacts of the detector of the device wafer 270, conductive contacts of the readout circuit 275, and / or metal bonds between conductive contacts of the detector and conductive contacts of the readout circuit 275. For example, the contacts 280 may include a contact layer formed on the detector to facilitate coupling with the readout circuit 275. In one embodiment, the device wafer 270 may be bump bonded to the readout circuit 275 using bonding bumps. The bonding bumps may be formed on the device wafer 270 and / or the readout circuit 275 to allow connection between the device wafer 270 and the readout circuit 275. In one aspect, hybridizing the device wafer 270 to the readout circuitry 275 may refer to bonding the device wafer 270 (e.g., the detector of the device wafer 270) to the readout circuitry 275 to mechanically and electrically combine the device wafer 270 and the readout circuitry 275. In one embodiment, a light shield may be coupled to a surface of the readout circuitry 275 to block stray light from being captured by the detector of the device wafer 270.

[0059] In one or more embodiments, various systems and methods are provided to facilitate monitoring the health of an imaging device. In one embodiment, the imaging device may be Figure 1 The imaging system 100 may include Figure 1 The imaging system 100 may be Figure 1 A portion of an imaging system 100. An imaging device may include an FPA that includes an active unit cell and a reference unit cell. Each active unit cell may include a detector. The detector of each active unit cell may capture a portion of an incident scene and generate an associated output. Each active unit cell has a structure (e.g., a shutter, a cover, a hood, a blade) or a portion thereof that selectively blocks the detector of the active unit cell from receiving a scene incident on the active unit cell. Each reference unit cell may include a detector and a structure (e.g., a shutter, a cover, a hood, a blade) or a portion thereof that blocks incident or external radiation (e.g., thermal radiation) from being received by the detector. In some cases, on a chip, a reference unit cell may be outside an active array including active unit cells and behave similarly to an active detector, except that the reference detector is shielded from the scene and is therefore effectively invisible to the scene. In an embodiment, the active unit cell and the reference unit cell include a bolometer (e.g., a microbolometer) as their detector.

[0060] In various embodiments, monitoring the health of an imaging device may be based on characteristics of an effective unit cell of an imaging device and / or characteristics of a reference unit cell of an imaging device. In some aspects, monitoring the health of an imaging device may be based on comparing characteristics of an effective unit cell with corresponding characteristics of a reference unit cell. Alternatively or additionally, monitoring the health of an imaging device may be based on comparing characteristics of an effective unit cell and / or a reference unit cell with corresponding characteristics previously determined and stored (e.g., during a calibration event performed at a factory or on-site). In some cases, an imaging device is part of a larger system (e.g., an ADAS). An imaging device may indicate (e.g., send a message, indicate, alert) the health of an imaging device or its components (e.g., a reference unit cell, an effective unit cell) to a system so that problems with an imaging device or its output (e.g., an image) may be solved, mitigated, and / or processed accordingly. As a non-limiting example, when a potential problem with an imaging device is identified, the system may perform mitigation actions, such as imaging a scene with a different imaging device, shutting down power to the imaging device until the imaging device can be repaired, ignoring images of a scene provided by the imaging device (e.g., because the image may be an inaccurate representation of the scene), and / or other actions.

[0061] For purposes of illustration, the imaging device is or includes an infrared imaging device, but the various embodiments may be applied to imaging devices for capturing images of bands outside the infrared band. With respect to the infrared imaging device, the effective unit cell may be sensitive to changes (e.g., small changes) in scene temperature. For example, for a bolometer, the resistance of the bolometer changes in response to changes in scene temperature. The resistance change of each bolometer can be converted into a time-division multiplexed electrical signal by the ROIC of the FPA. Various types of infrared detectors rely on high thermal isolation from the surrounding environment to achieve a desired performance level (e.g., a desired detector sensitivity). For example, in order to achieve a desired performance level, the effective unit cell is isolated from the ROIC coupled to the effective unit cell and / or the convective heat within the sensor cavity.

[0062] In various aspects, an FPA formed by a detector array (e.g., a bolometer array) and a ROIC can be located within a vacuum package to provide a high vacuum environment that helps achieve the desired sensitivity from the detector array. Due to various factors, such as degassing and / or leakage, vacuum integrity can degrade over time. For example, when vacuum is lost, air and / or other gases can enter the sensor cavity and adversely affect the thermal isolation of the detector. Such a reduction in vacuum integrity can lead to a gradual loss of vacuum pressure and thereby a reduction in the sensitivity and overall performance of the FPA. The reference detector of the FPA is shielded from incident or external radiation and, similar to the active detector, is substantially thermally isolated from the ROIC substrate. The active detector and the reference detector can share various properties, such as operating in the same vacuum cavity, temporal noise properties, and / or spatial noise properties. The sharing of such properties can allow the reference unit cell to be used for fault detection and compensation techniques. Therefore, using various embodiments, a comparison between the active unit cell and the reference unit cell can be used to monitor the health of the imaging device.

[0063] In some embodiments, vacuum integrity can be determined based on noise associated with an effective detector and noise associated with a reference detector. In this regard, the noise characteristics of the effective detector and the noise characteristics of the reference detector can be monitored and compared to determine vacuum integrity, and thereby determine the health of an imaging device including an effective detector and a reference detector. In some aspects, noise can be characterized (e.g., quantified) by measuring power spectral density (PSD). PSD can include fixed pattern noise (FPN) and time-varying random noise (RND). In terms of noise characteristics, a reduction or loss of vacuum reduces or eliminates the thermal isolation of the detector because the detector is thermally coupled to the ambient cavity, resulting in a more uniform output of each detector and lower temporal RND noise as the time constant increases. When it is determined that the vacuum level associated with the effective detector and / or the reference detector has dropped below a threshold, there may be a fault associated with the effective detector and / or the reference detector. In other cases, other ways of characterizing noise can be utilized. The PSD and / or other characteristics of noise can be determined during calibration.

[0064] In one aspect, each detector may be used to capture frames and these frames are used to calculate / characterize the PSD of the detector. The FPN and RND components of the PSD may be characterized in various ways. In one instance, a two-dimensional (2D) discrete cosine transform (DCT) may be used to characterize the FPN component and the RND component. As an example, Figure 3A , 3B and 3C use a 2D DCT of size 8×8 to compute / characterize the PSD of FPN, RND, and a weighted sum (e.g., mixture) of FPN and RND. Figure 3C Medium, γ PSD =0.5.

[0065] In some embodiments, to obtain FPN and RND, a shutter event may be performed in which a shutter or similar structure is used to block an active detector (e.g., an active bolometer) from receiving scene content. Thus, during the shutter event, the active detector is blocked from the scene, while the reference detector remains blocked from the scene. If desired, FPN and RND may be determined at calibration and refined during each shutter event. For a given detector, once the PSD of the detector (e.g., Figures 3A-3C 8×8 matrix in the example) is determined at the factory or otherwise in a known detector health state, the characteristics of the detector will generally not change as long as the detector is in the same health state as the detector was characterized. In some cases, the PSD of the detector may be stored to provide a known health state of the detector for comparison during runtime of the detector to determine during runtime whether the detector is still in a healthy state. In this regard, at a given moment during runtime, the PSD may be determined and compared to one or more historical performances determined from calibration to determine the health of the detector. In some cases, a shutter event may be part of a larger calibration event. In some cases, the PSD of each detector may be determined prior to leaving the factory. For example, prior to leaving the factory, the detector may be used to capture frames and calculate / characterize the PSD of the detector based on the captured frames. If desired, the PSD of each detector may be subsequently measured and improved during each shutter event (e.g., performed in the field).

[0066] The PSD can be determined during normal operation (e.g., when structures are not blocking the active detector from receiving the scene). When imaging a real-world scene, the PSD of the active detector is different from the PSD of the active detector when imaging a uniform scene (e.g., a scene provided during a calibration event or shutter event). While the active detector is imaging the real scene, the blind detector is still blocked from receiving the scene by the structure and is therefore invisible to the scene. At this point, the PSD of the reference detector can be used to represent a known state of a uniform image. The PSD of the active detector varies in magnitude over the operating temperature range as the background signal varies and as the noise varies with the magnitude of the signal. The PSD of the reference detector can be calculated on each frame to normalize the PSD of the active detector. The PSD of the reference detector can provide a calibration of the noise characteristics of the active detector over temperature on the fly (e.g., during field run time).

[0067] In some embodiments, the PSD associated with the active unit cell and the reference unit cell can be utilized in real time and compared to historical PSD values ​​associated with the imaging device to indicate the health of the imaging device. In some aspects, the health of the imaging device (e.g., the health of the active unit cell and / or the reference unit cell) can be determined based on a comparison of the PSD associated with the active unit cell and the PSD associated with the reference cell.

[0068] In a first example of a potential failure mode, the effective detector has a PSD that is the same as (e.g., matches) or sufficiently similar to the PSD of the reference detector when the effective detector should be imaging a real scene (e.g., not a uniform scene for calibration purposes). Such a situation may indicate an indication that the structure is stuck (e.g., the shutter cannot be moved out of the field of view of the effective detector), the optical device is blocked, and / or an indication of a poor signal from the scene. In the case where the imaging device is part of a larger system (e.g., ADAS), the imaging device may indicate to the system (e.g., send a message, indicate, alert) that the quality of the scene and / or the image of the scene captured by the imaging device may be poor, and / or there may be a problem with the effective detector and / or the reference detector. As an example, in one aspect, the shutter event can be used to determine the PSD of pure noise, wherein the pure noise PSD is measured at calibration and improved during each shutter event if necessary. If the captured signal is indistinguishable from pure noise, it is possible that the shutter is stuck in a closed state, the aperture is blocked, the vacuum is lost, the lens is broken, the focus is not correct, there are weather conditions that are not conducive to imaging (e.g., extreme humidity), or the effective detector is essentially blind.

[0069] In contrast, when a structure (e.g., shutter, uniform blade) is closed over the effective detector, the effective detector can be expected to have characteristics similar to the calibration and within the measurement error of the reference detector. In a second example of a failure mode, during a calibration event and / or a shutter event, the PSD of the effective detector is different from the PSD of the reference detector. In this case, the structure may be stuck (e.g., the shutter may not be closed over the effective detector). As an example of utilizing a reference detector, since the reference detector does not respond to changes in the scene, the offset in the reference detector tracks the temperature of the FPA. In some cases, if the imaging device uses an internal shutter (e.g., between the optics and the FPA), the temperature of the structure can also be approximated as the temperature of the FPA. During a shutter event, the offset from the reference detector can be compared to the effective detector and the averages of the two fields compared. In a normal state (e.g., a non-faulty state), the averages of the two fields are typically similar. If the change in the two fields exceeds a threshold amount, the shutter of the effective unit cell may be stuck in an open state (e.g., not closed over the effective detector). Where the imaging device is part of a larger system (eg, ADAS), the imaging device may indicate (eg, send a message, indicate, alert) to the system that there may be a problem with the active detector and / or the reference detector.

[0070] Typically, the response of the detector to a signal change (e.g., a change in IR radiation) is not instantaneous. In the case where the detector is a bolometer, the response of the detector can be a change in resistance in response to a change in IR radiation received by the detector. As an example, in a bolometer array, each active bolometer is configured to receive external IR radiation and is substantially thermally isolated from the ROIC substrate. The bolometer resistance changes with temperature changes, wherein the resistance change is measured by the ROIC and converted into a signal. The bolometer array is typically sensitive to small changes in temperature, and such sensitivity is facilitated by thermal isolation of the bolometer from the ROIC and / or convection within the sensor cavity.

[0071] As an example, Figure 4 A circuit diagram showing an example of a circuit for measuring the resistance of a microbolometer according to one or more embodiments of the present disclosure and a side view of a microbolometer are shown. Figure 4 The embodiments disclosed herein are not limiting, and may be applied to a wide variety of temperature sensing devices (e.g., infrared detectors) and associated circuit configurations that exhibit self-heating due to applied electrical bias. Thus, while a microbolometer may be discussed as a specific implementation example in one or more embodiments, it should be understood that the techniques disclosed herein may be applied to a wide variety of temperature sensing devices and associated configurations.

[0072] like Figure 4As shown, a voltage (denoted as V) is applied across a series combination of microbolometer 405 and resistive load 410. Microbolometer 405 (e.g., also referred to as an active microbolometer) is thermally isolated from substrate 415 and receives incident infrared radiation 420, while resistive load 410 may be a microbolometer (e.g., also referred to as a load microbolometer) that is thermally shorted to substrate 415. Substrate 415 may include an ROIC. As an example, microbolometer 405 (e.g., Figure 4 ) provides bridge 425 via legs 430, wherein legs 430 provide structural support, thermal isolation, and electrical connection to substrate 415 for bridge 425. The output voltage (denoted as V out ) to determine the resistance of microbolometer 405.

[0073] For a microbolometer, such as microbolometer 405, the resistance of the microbolometer changes over time. Therefore, the capture of an infrared image is based on the measurement of the resistance of the microbolometer. In order for the captured IR image to accurately represent the infrared radiation (e.g., 420) of the scene, the microbolometer is isolated from the ROIC and the convective heat within the sensor cavity.

[0074] In some embodiments, a thermal time constant (TTC) may be utilized to determine the health of an active unit cell and / or a reference unit cell. A characterization of the rate of change of a bolometer may be provided by the TTC of the bolometer. The TTC provides a measure of the amount of time required for an indication detector to change to a resistance level indicative of a signal received by the detector. When a transient signal is provided to a bolometer, the amount of time it takes for the bolometer resistance to change by (1-1 / e) (i.e., approximately 63%) is the TTC of the bolometer. For each successive TTC cycle, the resistance of the bolometer changes by another (1-1 / e) relative to the previous TTC cycle. As an example, the nominal TTC of a bolometer may be 8 milliseconds.

[0075] As an example, Figure 5 An example graph of the change in resistance of a bolometer as a function of time according to one or more embodiments of the present disclosure is shown. At time t=0, 100% of the signal is immediately provided to the microbolometer. After one TTC has passed (denoted as t=τ), the resistance of the microbolometer has changed by (1-1 / e) (e.g., about 63%). After another TTC has passed (denoted as t=2τ), the resistance of the bolometer has changed by another (1-1 / e) relative to the resistance at t=τ. In the graph, at t=τ, the change in detector resistance indicates about 63% of the signal received by the bolometer. At t=2τ, 3τ, and 4τ, the change in detector resistance indicates about 86.5%, 95.0%, and 98.2% of the signal received by the bolometer.

[0076] In one aspect, the TTC of a detector can be determined by measuring the self-heating due to pulsing the bias on the detector. The bias (e.g., bias voltage and / or bias duty cycle) can be changed to heat the effective bolometer beyond the normal self-heating of the bolometer and observe the temperature decay. The TTC can be determined from the temperature decay. A reference detector is not used to detect scene information, but can be used as a detector in the same vacuum as the effective detector and can be used to measure the TTC. The bias applied to the reference detector of the imaging device can be changed during operation of the imaging device so that the imaging device measures the TTC for each frame in real time and allows the TTC to determine whether the vacuum of the imaging device has been reduced or lost. In one aspect, a change in vacuum pressure is associated with a proportional change in TTC. Example technology for a pulse biased detector can be found in U.S. Patent No. 8,471,206, the entire contents of which are incorporated herein by reference.

[0077] In an embodiment, to measure the health of an imaging device, the TTC and / or vacuum integrity of a reference detector may be evaluated at a frame rate by reading the reference detector simultaneously with the effective detector at the frame rate. In another embodiment, the TTC and / or vacuum integrity may be evaluated at a faster rate than the frame rate. In some aspects, the positioning of the reference unit cell relative to the effective unit cell may affect the rate at which the TTC and / or vacuum integrity may be evaluated. In the case of a row-based readout architecture, a pulse biasing technique may be applied to each row of blind detectors at a time. A host (e.g., an ADAS or its user) may monitor the output of the reference detector and determine whether the temperature decay is within the nominal operating range to determine whether the remainder of the frame data has been captured in a valid state. When it is determined that the imaging device is healthy, the frame data may be referred to as being captured in a valid state. For an unhealthy imaging device, the frame data from the imaging device may be an inaccurate representation of the scene.

[0078] A reference unit cell may be provided in one or more locations of the imaging device. As an example, Figure 6 An example array 600 (e.g., also referred to as an active array) and potential locations 605, 610, 615, and 620 of one or more reference unit cells are shown in accordance with one or more embodiments of the present disclosure. The active unit cells of the array 600 are generally sensitive to changes (e.g., small changes) in scene temperature. To facilitate such sensitivity, the array 600 can be thermally isolated from, for example, convection within a ROIC and / or sensor cavity coupled to the array 600. In some embodiments, thermal isolation can be achieved by the detector being mechanically elevated on a table-like structure (e.g., a countertop-like structure) by thin legs on the ROIC and being encapsulated in a vacuum to prevent thermal convection that transfers heat through gas particles. In an embodiment, the array 600 and potential locations 605, 610, 615, and 620 can collectively provide Figure 2A The unit cell array 205 includes Figure 2A The unit cell array 205, as Figure 2A 605, 610, 615, and / or 620, or any of the potential positions 605, 610, 615, and 620, do not include one or more reference unit cells. Also note that the potential positions 605, 610, 615, and / or 620 are provided as non-limiting examples. Alternatively or additionally, one or more reference unit cells may be disposed at another position.

[0079] Potential location 605 is independent and remote from array 600. In this case, a pulse bias method can be used with array 600 and one or more reference unit cells at potential location 605 to determine vacuum health at a frame rate. Potential location 610 represents one or more rows of reference unit cells above and / or below array 600. The rows of reference unit cells share a row-based readout architecture with array 600 (e.g., the active unit cells of array 600). Similar to setting the reference unit cell at potential location 605, the rows of reference unit cells will also be restricted to readout at a frame rate. In one case, potential location 620 can include a reference unit cell that is read out together with the reference unit cell at potential location 610.

[0080] Potential location 615 represents one or more columns of reference unit cells on the left and / or right side of array 600. The columns of reference unit cells share a row-based readout architecture and can be read out at the end of each row. In one case, potential location 620 can include a reference unit cell that is read out along with the reference unit cell at potential location 615. By using such a reference column and measuring the health of the imaging device (e.g., vacuum integrity of the imaging device) at the row rate used for the pulse bias method (e.g., the frame rate divided by the number of rows of array 600), the amount of time to detect a fault (e.g., FTDI in an ADAS application) can occur faster relative to the case where the reference unit cells are read out at the frame rate.

[0081] Note that the foregoing is with respect to a row-based readout architecture. In a column-based readout architecture, potential location 610 may include one or more columns of reference unit cells to the left and / or right of array 600, wherein array 600 and the reference unit cells share the column-based readout architecture. When utilizing the reference unit cells at potential location 610, detection of a fault may occur at a frame rate. Potential location 615 may include one or more rows of reference unit cells above and / or below array 600. In this case, the health of the imaging device may be determined at a column rate (e.g., the frame rate divided by the number of columns of array 600) based on reading out blind rows.

[0082] In some aspects, a host (e.g., an ADAS or its user) evaluates various valid and / or reference unit cells. In some aspects, an evaluation of a reference detector may be performed by the ROIC to compare the decay rate of the reference detector to a circuit with a known decay rate. In some cases, such an evaluation may be provided as a real-time test mode and / or by the ROIC. The reference detector may be pulse biased, and the output of the reference detector may be passed through a comparator.

[0083] As an example, a reference detector may be pulse biased and compared to a first threshold decay rate, such as 6ms. In this example, if the TTC is above the first threshold decay rate, an error bit (e.g., an error flag) may be set in the output data stream. Alternatively or additionally, the reference detector may be compared to a second threshold decay rate (e.g., 10ms). If the TTC is below the second threshold decay rate, a separate error bit may be set in the output data structure. The attenuation of the blind detector may be compared to reference attenuations of 6ms and 10ms. If the attenuation exceeds the reference attenuation, an error bit may be set in the output data of the sensor. When performing such an evaluation at the ROIC (or other component of the imaging device), the host may not have to compare real-time data from the reference detector, but instead look up a status bit (e.g., an error bit status) from the ROIC (or other component of the imaging device), which may reduce fault detection time (e.g., FTDI) by reducing processing time at the host (e.g., host ADAS system).

[0084] In general, it is important to know the validity of the data (e.g., of an imaging device) for each data transmission. In various embodiments, techniques are provided to facilitate the validity of such data. In an example automotive application, data transmission can be from a detector (e.g., a bolometer) to image processing electronics, an electronic control unit (ECU), a final ADAS decision maker (e.g., for controlling the navigation of the car), and / or any intermediate data switching. Table 1 provides an example image frame format for facilitating data validation between an imaging device and an ECU.

[0085] Table 1: Image frame format

[0086]

[0087] Wherein, FC represents a frame counter, LC represents a line counter, STAT represents status data, TELE represents a telemetry line, VIDEOx represents a line of image / video data from an imaging device, where x=0, 1, ..., 511, CRC represents a checksum (e.g., a 32-bit checksum), and PAD represents padding bits (e.g., 32 bits of zero to preserve an 8-byte boundary scheme). As shown in Table 1, lines 0 and 1 of an image frame include telemetry data, and lines 2 to 513 include image / video data. For example, line 2 of an image frame includes image / video data from a detector in row 0 of a detector array, and line 513 of an image frame includes image / video data from a detector in row 511 of the detector array.

[0088] In some embodiments, tests and metadata from the detector may be provided (e.g., injected) in the image frame to facilitate monitoring of the health of the imaging device. Such data and its verification may be used to determine whether there are lost frames, delays, latency, and / or varying frame rates. Data and its verification may be used to ensure that the imaging device and components downstream of the imaging device (e.g., other components of the ADAS system) are processing the correct data at a given time (e.g., not outdated data or otherwise unsynchronized data). The imaging device may count each detector in the row while being read out by the ROIC. Such counter values ​​may be referred to as detector counters / digits or pixel counters / digits. Such counter values ​​may be attached to rows (e.g., data of image frames associated with rows) as well as current frame counters / digits, current line counters / digits (e.g., indicating current row counters / digits) and metadata. When a line of an image frame (e.g., associated with a row) is completed, a hash value may be calculated for the line and attached to the line. In one aspect, the hash may be a cyclic redundancy check (CRC). A hash value may be calculated for each row of the detector array of the imaging device and embedded in the data output (e.g., image frame).

[0089] As an example, Figure 7 An example image frame format 700 according to one or more embodiments of the present disclosure is shown. The image frame format may be used in automotive applications. An image frame having the image frame format 700 includes lines (e.g., ECU telemetry lines or camera lines), wherein each line includes a frame counter (FC), a line counter (LC), a pixel counter (PC), metadata (Meta), and hash data (HASH). The image frame format 700 includes instructions for the ECU, the imaging device (in Figure 7 ) and the detector of the imaging device (in Figure 7The FC, LC, PC, metadata, and hashed data of the ECU (represented as sensor in the example). Meta data may include, for example, input counters and / or timestamps from further upstream in the data path. A hash value (e.g., CRC value) is determined for each line, represented as an ECU line hash and a camera line hash. Note that Figure 7 An example image frame format is provided. The fields of the image frame format 700 may be Figure 7 The fields shown are reordered. Other image frame formats may include additional, fewer and / or different fields.

[0090] By including a corresponding hash value (e.g., CRC) for each line of the image frame, the imaging device can calculate an expected hash value for each line (e.g., associated with the row) of the image frame (based on the data embedded in the image frame for each row excluding the hash value itself) and compare the expected hash value to the embedded hash value to determine the validity of the data transmission from the imaging device. The hash values ​​can be used to verify the quality of the various counters / numbers embedded in the image frame. The frame counters provided in each line of the image frame can be used to verify that the data stream is continuous, the frames are received in the correct order, and no data is lost or created. Verification can also be performed to determine that the correct number of rows are present, the order of the rows is correct, and the correct number of detectors are present in each row.

[0091] In some aspects, the metadata embedded in the image frame includes a timestamp. The timestamp provides a relative system clock that can be used by an imaging device that operates according to an external system clock to allow verification of delays and wait times between frames from the imaging device. Such verification can allow accurate synchronization of the imaging device with other devices (e.g., sensors) of a larger system (e.g., an automotive system), and can allow accurate determination of when data from the imaging device is acquired, delays and / or wait times from the imaging device. Such verification contrasts with conventional methods. In conventional methods, the imaging device receives synchronization pulses and generates frames in response to the received synchronization pulses, but the imaging device cannot determine in time which pulse corresponds to which image frame.

[0092] In one aspect, a timestamp and / or counter may be provided to the ROIC (and / or other camera electronics) for encoding in the sensor output by the ROIC (and / or other ROIC electronics). A counter may be utilized to facilitate monitoring of the timing of a frame, and a hash value may be utilized to verify the counter. In some cases, a timestamp tag or frame counter may be provided to the ROIC or other camera electronics to help ensure that there are no dropped frames in the data provided to the host. In some cases, a timestamp tag / counter and CRC may be provided to the host as metadata (e.g., telemetry data) to help ensure / verify system health. In some cases, sensor-level timestamps (e.g., per line or per frame) may be used to synchronize with other sensors. Detection of lost frames, latency, frame rate variations, and unexpected delays may be facilitated. The system may determine that it is operating on the correct data at the correct time, which helps enhance safety. In one aspect, the head unit or central processor of the car may provide a counter or timestamp to the imaging device. The imaging device may perform a CRC on the input or otherwise mark the input and embed the CRC or other mark in the data output by the imaging device. Such a host provided input may be the same counter provided to one or more other sensors, which then allows the host or car to determine delays and latencies through the entire system by utilizing sensors and imaging devices.

[0093] In some embodiments, a known digital pattern may be provided in an image frame (e.g., injected into the image frame) to facilitate verification of the health of the imaging device. The digital pattern may be embedded in the pixel data. The pattern may be embedded in the data from valid pixels and / or reference pixels. Such a pattern may be analyzed and subsequently removed by the camera electronics. On the one hand, the pattern may be used to verify the health of the imaging device and to alert the host when the data is corrupted. In this regard, the imaging device knows the pattern superimposed on the pixel data and is expected to be able to remove the pattern from the data before using (e.g., being provided for processing, storage and / or display). For example, the camera electronics may monitor the pattern and alert the host when the pattern changes. In this example, if a portion of the pattern remains in the image after extraction, it may be determined that there is a problem with the sensor. On the one hand, such a digital pattern may be referred to as a digital watermark.

[0094] In one aspect, such a digital pattern can be provided after the ADC. After the ADC, the sensor can overlay the pattern onto the image data. The pattern can be a static or changing pattern. As an example, the pattern can be in a fixed position but change over time (e.g., like a timestamp). As another example, the pattern can be a fixed pattern that changes position with each frame. For example, the pattern can slide along each frame so that for each frame, the pattern moves one pixel relative to the previous frame.

[0095] Figure 8 A flowchart of an example of a process 800 for facilitating monitoring the health of an imaging device according to one or more embodiments of the present disclosure is shown. For purposes of explanation, reference is made herein to Figure 1 The process 800 is described with reference to the imaging system 100; however, the process 800 is not limited to Figure 1 Various components of the imaging system 100. Note that one or more operations may be combined, omitted, and / or performed in a different order as desired.

[0096] At block 805, a characteristic of at least one active unit cell of an FPA (e.g., an FPA of imaging system 100) and / or a characteristic of at least one reference unit cell of the FPA (e.g., by imaging system 100) is determined. In some cases, the characteristic may be associated with the vacuum integrity of the FPA. As non-limiting examples, the characteristic may be a PSD (e.g., FPN and / or RND components), a thermal time constant, and a temperature decay rate.

[0097] At block 810, a state of the FPA is determined (e.g., by the imaging system 100) based at least in part on the characteristic. As one example, the state of the FPA may be determined based on a comparison of a characteristic of the active unit cell (e.g., PSD and / or vacuum integrity) with a corresponding characteristic of a reference unit cell. As another example, the state of the FPA may be determined based on a comparison of a characteristic of the active unit cell (e.g., PSD) with a previously determined and stored corresponding characteristic (e.g., a PSD determined during a previous calibration event and stored for future use). In one aspect, the state of the FPA may be a healthy state or an unhealthy state.

[0098] At box 815, an indication of the state of the FPA is transmitted. As an example, the indication may be transmitted to another portion of the imaging system 100 that includes the FPA and / or a larger system that includes the imaging system 100, such as an ADAS. In some cases, an indication of a healthy state may be transmitted to verify that the FPA is determined to be operating correctly. In some cases, an unhealthy indication may indicate that mitigation actions may need to be performed on the FPA. When potential problems with the FPA are identified and marked by the indication, mitigation actions may be performed. As non-limiting examples, mitigation actions may include imaging the scene using a different imaging device, turning off power to the imaging device until the imaging device can be repaired, ignoring images of the scene provided by the imaging device (e.g., because the image may be an inaccurate representation of the scene), and / or other actions.

[0099] Fig. 9 900 is a flowchart showing an example of a process 900 for generating an image according to one or more embodiments of the present disclosure. For the purpose of explanation, reference is made here to Figure 1The process 900 is described with reference to the imaging system 100; however, the process 900 is not limited to Figure 1 Various components of the imaging system 100. Note that one or more operations may be combined, omitted, and / or performed in a different order as desired.

[0100] At block 905, image data associated with scene 160 is captured (e.g., by image detector circuitry 165). At block 910, a hash value is generated based at least on the image data (e.g., by readout circuitry 170 or other component). In some cases, at least a subset of the hash values ​​are generated based further on respective detector identifiers associated with detectors of image detector circuitry 165 that generated respective portions of the image data, a current frame number, a current line number, a timestamp associated with a time of capture of the image data, and / or a data pattern (e.g., a watermark). Where image detector circuitry 165 includes an array of detectors, rows of the array may be associated with lines of the image. At block 915, the image is transmitted. The image includes the image data, the hash value, and other data used to generate the hash value (e.g., current frame number, current line number).

[0101] Fig.10 A flowchart of an example of a process 1000 for determining the validity of a received image according to one or more embodiments of the present disclosure is shown. Note that one or more operations may be combined, omitted, and / or performed in a different order as desired. In an embodiment, process 1000 may be performed by an ADAS system that includes or is otherwise coupled to imaging system 100. At box 1005, an image including data and a first hash value is received (e.g., by an ADAS system). The data may include image data and other data, such as a detector identifier, a current frame number, a current line number, a timestamp associated with the capture time of the image data, and / or a data pattern (e.g., a watermark). At box 1010, a second hash value is generated based on the data. At box 1015, the validity of the image is determined based on the first hash value and the second hash value. If the first hash value matches the second hash value, it may be determined that the image is valid and the image data and / or other data contained therein may be used (e.g., by an ADAS system). If the first hash value does not match the second hash value, it may be determined that the image is invalid. When it is determined that the image is invalid, subsequent processing may be performed to determine whether the data contained in the image is available. Alternatively or additionally, troubleshooting and / or mitigation actions (eg, repairs) may be performed on the imaging system 100 that generated the image.

[0102] Where applicable, hardware, software, or a combination of hardware and software may be used to implement the various embodiments provided by the present disclosure. Also where applicable, without departing from the spirit of the present disclosure, the various hardware components and / or software components set forth herein may be combined into composite components comprising software, hardware, and / or both. Where applicable, without departing from the spirit of the present disclosure, the various hardware components and / or software components set forth herein may be divided into subcomponents comprising software, hardware, or both. In addition, where applicable, it may be expected that software components may be implemented as hardware components, and vice versa.

[0103] Software according to the present disclosure, such as non-transitory instructions, program code and / or data, can be stored on one or more non-transitory machine-readable media. It is also contemplated that the software identified herein can be implemented using one or more general-purpose or special-purpose computers and / or networked and / or non-networked computer systems. Where applicable, the order of the various steps described herein can be changed, combined into composite steps and / or divided into sub-steps to provide the features described herein.

[0104] The above description is not intended to limit the present disclosure to the precise form disclosed or the specific field of use. The above embodiments illustrate but do not limit the present invention. It is contemplated that various alternative embodiments and / or modifications of the present invention, whether explicitly described or implied herein, are possible. Therefore, the scope of the present invention is limited only by the appended claims.

Claims

1. A method, comprising: determining one or more characteristics of at least one active unit cell of a focal plane array (FPA) of an imaging system and at least one reference unit cell of the FPA, wherein the at least one active unit cell comprises a detector selectively shielded from an incident scene by a structure, wherein the at least one reference unit cell comprises a reference detector shielded from the incident scene, wherein the one or more characteristics are based on a power spectral density (PSD) associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell; determining a state of the FPA based at least in part on a PSD associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell; determining a state of the structure based at least in part on the PSD associated with the at least one active unit cell and the PSD associated with the at least one reference unit cell; and An indication of a status of the imaging system is transmitted to selectively cause adjustment of the imaging system, wherein the indication is based on a status of the structure and a status of the FPA.

2. The method according to claim 1, wherein: The indication is transmitted as part of an image comprising image data indicative of the incident scene.

3. The method according to claim 1, wherein: The indication is transmitted to an advanced driver assistance system to facilitate operation of a vehicle including the FPA.

4. The method according to claim 1, further comprising: generating a hash value based at least in part on data associated with the incident scene; generating an image including the data and the hash value; and The image is transmitted.

5. The method according to claim 4, wherein: The data includes a detector identifier associated with the at least one active unit cell, a frame number, a line number associated with the at least one active unit cell, a timestamp, and / or a watermark.

6. The method according to claim 1, wherein: The one or more characteristics are indicative of a vacuum integrity associated with the at least one active unit cell and / or a vacuum integrity associated with the at least one reference unit cell.

7. The method of claim 1 , further comprising determining a thermal time constant associated with the at least one reference unit cell, wherein: The state of the FPA is also based on the thermal time constant.

8. The method according to claim 1, further comprising: applying a pulse bias signal to the at least one reference unit cell; and determining a temperature decay rate associated with the at least one reference unit cell after the applying, Wherein, the state of the FPA is also based at least in part on the temperature decay rate.

9. The method according to claim 8, wherein: The temperature decay rate is determined by the readout circuitry of the FPA.

10. The method of claim 8, further comprising determining a thermal time constant associated with the at least one reference unit cell based on the temperature decay rate.

11. The method according to claim 8, wherein: The state of the FPA is based on whether the temperature decay rate is below a first threshold, above a second threshold, and / or between the first threshold and the second threshold.

12. The method according to claim 1, wherein: The one or more characteristics include a difference between a PSD associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell when a detector of the at least one active unit cell is shielded from the incident scene.

13. The method according to claim 1, wherein: The one or more characteristics include a difference between a PSD associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell when a detector of the at least one active unit cell is not shielded from the incident scene.

14. The method of claim 1, further comprising determining a PSD associated with the at least one active unit cell when a detector of the at least one active unit cell is not shielded from the incident scene, and wherein, The characteristic comprises a difference between a PSD associated with the at least one active unit cell and a stored PSD determined during calibration of the FPA.

15. The method according to claim 1, wherein: The one or more characteristics are determined by readout circuitry.

16. The method of claim 1, wherein: Each active unit cell includes a detector that is selectively shielded from an incident scene by a structure and selectively captures the incident scene; and Each reference unit cell includes a detector and a structure that blocks the incident scene from being received by the detector of the reference unit cell.

17. The method according to claim 1, further comprising: receiving an image including data and a first hash value, wherein the data includes a detector identifier associated with an active unit cell, a frame number, a line number associated with the active unit cell, a timestamp, and / or a watermark; generating a second hash value based on the data; and determining the presence of lost frames, delay, latency, and / or a non-constant frame rate based at least on the data, the first hash value, and the second hash value; and The validity of the image is determined based at least on the presence of lost frames, delays, latency, and / or a non-constant frame rate.

18. An imaging system, comprising: A focal plane array (FPA), the focal plane array comprising: a plurality of active unit cells, wherein each active unit cell comprises a detector configured to be selectively shielded from an incident scene by the structure and configured to selectively generate an output in response to the incident scene; a plurality of reference unit cells, wherein each reference unit cell comprises a detector that is shielded from an incident scene; and A processing circuit, the processing circuit being configured to: determining one or more characteristics of at least one active unit cell of the plurality of active unit cells and / or at least one reference unit cell of the plurality of reference unit cells, wherein the one or more characteristics are based on a power spectral density (PSD) associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell; determining a state of the FPA based at least in part on a PSD associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell; determining a state of the structure based at least in part on the PSD associated with the at least one active unit cell and the PSD associated with the at least one reference unit cell; and An indication of a status of the imaging system is transmitted to selectively cause adjustment of the imaging system, wherein the indication is based on a status of the structure and a status of the FPA.

19. The imaging system of claim 18, wherein: The plurality of active unit cells are arranged in columns and rows of an array, and wherein the FPA further comprises a readout circuit configured to read each row of the array simultaneously with a corresponding one or more of the plurality of reference unit cells.

20. The imaging system of claim 18, wherein: The indication is transmitted to an advanced driver assistance system to facilitate operation of a vehicle including the imaging system.

21. The imaging system of claim 18, wherein: The processing circuit is further configured to: generating a hash value based at least in part on data associated with the incident scene; generating an image including the data and the hash value; and The image is transmitted.

22. The imaging system of claim 21, wherein: The data includes a detector identifier associated with the at least one active unit cell, a frame number, a line number associated with the at least one active unit cell, a timestamp, and / or a watermark.

23. The imaging system of claim 18, wherein: The one or more characteristics are indicative of a vacuum integrity associated with the at least one active unit cell and / or a vacuum integrity associated with the at least one reference unit cell.

24. The imaging system of claim 18, wherein: The processing circuit is further configured to determine a thermal time constant associated with the at least one reference unit cell, wherein the state of the FPA is further based on the thermal time constant.

25. The imaging system of claim 18, wherein: The processing circuit is configured to determine a temperature decay rate associated with the at least one reference unit cell in response to a pulsed bias signal applied to the at least one reference unit cell, and wherein a state of the FPA is also based at least in part on the temperature decay rate.

26. The imaging system of claim 18, wherein: The one or more characteristics include a difference between a PSD associated with the at least one active unit cell and a PSD associated with the at least one reference unit cell.

27. The imaging system of claim 18, wherein: The processing circuit is further configured to determine a PSD associated with the at least one active unit cell when a detector of the at least one active unit cell is not shielded from the incident scene, and wherein the characteristic comprises a difference between the PSD and a stored PSD determined during calibration of the FPA.

28. The imaging system of claim 18, wherein: The FPA further comprises a readout circuit, and wherein the readout circuit comprises the processing circuit.

Images